diff --git a/README.md b/README.md index a8b5a69..7350316 100644 --- a/README.md +++ b/README.md @@ -28,11 +28,17 @@ For detailed deployment options (bare Node.js, reverse proxy, backups), see the - Register services with health check endpoints and poll them on configurable intervals (5s to 1hr) - Exponential backoff on failures with circuit breaker protection (opens after 10 consecutive failures) - Custom schema mapping for non-standard health endpoints, including object-keyed formats (Spring Boot Actuator, ASP.NET Health Checks, etc.) with skipped-check support +- **OTLP push ingestion** — receive metrics via `POST /v1/metrics` and traces via `POST /v1/traces` from OpenTelemetry collectors with team-scoped API key authentication, auto-registration of unknown services, and per-service custom metric/attribute name mappings +- **OTLP trace-based dependency discovery** — automatically discover dependencies from CLIENT and PRODUCER spans, with full span storage for future trace timeline views, configurable retention (default 7 days), and auto-association to registered services +- **Histogram and sum metric processing** — extract percentile latency (p50/p95/p99) from OTLP histogram metrics and request counts from sum metrics via linear interpolation +- **Prometheus scraping** — poll Prometheus text exposition endpoints (`text/plain; version=0.0.4`) with automatic metric-to-dependency mapping and per-service custom metric/label name mappings - Contact info and impact overrides with 3-tier merge hierarchy (instance > canonical > polled) — resolved in API responses - Per-hostname concurrency limiting and request deduplication prevent polling abuse **Visualization** - Interactive dependency graph (React Flow) with team filtering, search, layout controls, automatic high-latency detection, and isolated tree view +- Visual distinction for auto-discovered vs manually-configured dependencies (dashed edges with "Suggested" badge for unconfirmed trace-discovered associations) +- Org-wide external node enrichment — add display names, descriptions, impact, and contact info to shared external dependencies (e.g., Stripe, PostgreSQL) - Latency charts (min/avg/max over time) and health timeline swimlanes per dependency - Edge selection shows per-dependency latency chart, contact info, impact, and error history - Node selection shows aggregate latency chart across all dependents and merged contact info @@ -77,8 +83,8 @@ For detailed deployment options (bare Node.js, reverse proxy, backups), see the **Operations** - SQLite database — zero external dependencies, sessions survive restarts -- Automatic data retention cleanup (configurable period, default 365 days) -- Runtime-configurable admin settings (retention, polling, rate limits, alerts) +- Automatic data retention cleanup (configurable period, default 365 days) with separate span retention (default 7 days, admin-configurable) +- Runtime-configurable admin settings (retention, span retention, polling, rate limits, alerts) - Structured JSON logging in production via pino - Docker image with health check and volume-mounted data @@ -193,8 +199,10 @@ cp server/.env.example server/.env | `SSL_CERT_PATH` | — | PEM certificate path (pair with `SSL_KEY_PATH`) | | `SSL_KEY_PATH` | — | PEM private key path (pair with `SSL_CERT_PATH`) | | `HTTP_PORT` | — | Plain HTTP port for health checks + redirect when `ENABLE_HTTPS=true` | -| `RATE_LIMIT_MAX` | `100` | Max requests per IP per 15-minute window | -| `AUTH_RATE_LIMIT_MAX` | `10` | Max auth requests per IP per minute | +| `RATE_LIMIT_MAX` | `3000` | Max requests per IP per minute (global) | +| `AUTH_RATE_LIMIT_MAX` | `20` | Max auth requests per IP per minute | +| `OTLP_RATE_LIMIT_MAX` | `600` | Max OTLP requests per IP per minute (global) | +| `OTLP_PER_KEY_RATE_LIMIT_RPM` | `150000` | Default per-API-key rate limit (requests/minute) | **Operations:** @@ -308,23 +316,28 @@ All endpoints require authentication unless noted. Admin endpoints require the a | Users | CRUD on `/api/users` (admin), `POST` and `PUT /:id/password` (local auth) | | Aliases | CRUD on `/api/aliases` (admin for mutations), `GET /canonical-names` | | Overrides | `GET/PUT/DELETE /api/canonical-overrides/:name`, `PUT/DELETE /api/dependencies/:id/overrides` | -| Associations | CRUD on `/api/dependencies/:id/associations` | +| Associations | CRUD on `/api/dependencies/:id/associations`, `PUT /:assocId/confirm`, `PUT /:assocId/dismiss` | +| Discovered Deps | `GET /api/services/:id/dependencies/discovered`, `PATCH /api/dependencies/:id/enrich` | +| External Nodes | `GET/PUT/DELETE /api/external-nodes/:canonicalName` | | Graph | `GET /api/graph` with `team`, `service`, `dependency` filters | | History | `GET /api/latency/:id` + `/buckets`, `GET /api/errors/:id`, `GET /api/dependencies/:id/timeline`, `GET /api/services/:id/poll-history` | -| Admin | `GET/PUT /api/admin/settings`, `GET /api/admin/audit-log` | +| Admin | `GET/PUT /api/admin/settings`, `GET/PUT /api/admin/settings/span-retention`, `GET /api/admin/audit-log` | | Manifest | `GET/POST /api/teams/:id/manifests`, `GET/PUT/DELETE /:id/manifests/:configId`, `POST /:id/manifests/sync`, `POST /:id/manifests/:configId/sync`, `GET /:id/manifests/:configId/sync-history`, `POST /api/manifest/validate`, `POST /api/manifest/test-url` | | Drift Flags | `GET /api/teams/:id/drifts` + `/summary`, `PUT /:driftId/accept` + `/dismiss` + `/reopen`, `POST /bulk-accept` + `/bulk-dismiss` | | Catalog | `GET /api/catalog/external-dependencies` — canonical name registry with team usage, descriptions, and aliases | | Alerts | CRUD on `/api/teams/:id/alert-channels` + `/test`, `GET/PUT /:id/alert-rules`, `GET /:id/alert-history`, `GET/POST /:id/alert-mutes`, `DELETE /:id/alert-mutes/:muteId`, `GET /api/admin/alert-mutes` | +| OTLP | `POST /v1/metrics`, `POST /v1/traces` (API key auth, OTLP JSON) | +| API Keys | `GET/POST /api/teams/:id/api-keys`, `DELETE /:id/api-keys/:keyId` | ## Security Depsera includes defense-in-depth security: - **Security headers** via Helmet (CSP, HSTS, X-Frame-Options, X-Content-Type-Options) +- **API key authentication** for OTLP push endpoints — team-scoped keys with SHA-256 hashing, prefix display, and last-used tracking - **SSRF protection** on health endpoints with private IP blocking and DNS rebinding prevention; configurable allowlist for internal networks - **CSRF protection** via double-submit cookie pattern -- **Rate limiting** — global (100 req/15min) and auth-specific (10 req/min) per IP +- **Rate limiting** — global (3000 req/min), auth-specific (20 req/min), OTLP global (600 req/min per IP), and per-API-key token bucket (150k req/min default) - **Session secret validation** — production startup refuses weak or missing secrets - **Redirect validation** prevents open redirect attacks on logout - **Body size limit** (100KB) on JSON payloads diff --git a/client/src/App.test.tsx b/client/src/App.test.tsx index f3567f7..ca4796a 100644 --- a/client/src/App.test.tsx +++ b/client/src/App.test.tsx @@ -31,12 +31,12 @@ describe('App', () => { expect(await screen.findByText('Sign in to continue')).toBeInTheDocument(); }); - it('shows the dashboard title on login page', async () => { + it('shows the logo on login page', async () => { mockFetch .mockResolvedValueOnce({ ok: false, status: 401, json: () => Promise.resolve({}) }) .mockResolvedValueOnce({ ok: true, status: 200, json: () => Promise.resolve({ mode: 'oidc' }) }); renderApp(); - expect(await screen.findByRole('heading', { name: 'Depsera' })).toBeInTheDocument(); + expect(await screen.findByAltText('Depsera')).toBeInTheDocument(); }); }); diff --git a/client/src/App.tsx b/client/src/App.tsx index 546bb79..ca1d04f 100644 --- a/client/src/App.tsx +++ b/client/src/App.tsx @@ -17,6 +17,7 @@ import ServiceCatalog from './components/pages/Catalog/ServiceCatalog'; import ManifestPage from './components/pages/Manifest/ManifestPage'; import ManifestAdmin from './components/pages/Admin/ManifestAdmin'; import AlertMutesAdmin from './components/pages/Admin/AlertMutesAdmin'; +import OtlpAdmin from './components/pages/Admin/OtlpAdmin'; function App() { return ( @@ -80,6 +81,14 @@ function App() { } /> + + + + } + /> } /> diff --git a/client/src/api/apiKeys.ts b/client/src/api/apiKeys.ts new file mode 100644 index 0000000..d990a9e --- /dev/null +++ b/client/src/api/apiKeys.ts @@ -0,0 +1,50 @@ +import { handleResponse } from './common'; +import { withCsrfToken } from './csrf'; + +export interface ApiKey { + id: string; + team_id: string; + name: string; + key_prefix: string; + last_used_at: string | null; + created_at: string; + created_by: string; + rate_limit_rpm: number | null; + rate_limit_admin_locked: number; +} + +export interface ApiKeyWithRawKey extends ApiKey { + rawKey: string; +} + +export async function listApiKeys(teamId: string): Promise { + const response = await fetch(`/api/teams/${teamId}/api-keys`, { + credentials: 'include', + }); + return handleResponse(response); +} + +export async function createApiKey( + teamId: string, + name: string +): Promise { + const response = await fetch(`/api/teams/${teamId}/api-keys`, { + method: 'POST', + headers: withCsrfToken({ 'Content-Type': 'application/json' }), + body: JSON.stringify({ name }), + credentials: 'include', + }); + return handleResponse(response); +} + +export async function deleteApiKey(teamId: string, keyId: string): Promise { + const response = await fetch(`/api/teams/${teamId}/api-keys/${keyId}`, { + method: 'DELETE', + headers: withCsrfToken(), + credentials: 'include', + }); + if (!response.ok) { + const error = await response.json().catch(() => ({ message: 'Delete failed' })); + throw new Error(error.message || error.error || `HTTP error ${response.status}`); + } +} diff --git a/client/src/api/associations.ts b/client/src/api/associations.ts index e92ba21..1058ff9 100644 --- a/client/src/api/associations.ts +++ b/client/src/api/associations.ts @@ -35,3 +35,25 @@ export async function deleteAssociation( throw new Error(error.message || `HTTP error ${response.status}`); } } + +export async function confirmAssociation( + depId: string, + assocId: string, +): Promise<{ success: boolean }> { + const response = await fetch( + `/api/dependencies/${depId}/associations/${assocId}/confirm`, + { method: 'PUT', headers: withCsrfToken(), credentials: 'include' }, + ); + return handleResponse<{ success: boolean }>(response); +} + +export async function dismissAssociation( + depId: string, + assocId: string, +): Promise<{ success: boolean }> { + const response = await fetch( + `/api/dependencies/${depId}/associations/${assocId}/dismiss`, + { method: 'PUT', headers: withCsrfToken(), credentials: 'include' }, + ); + return handleResponse<{ success: boolean }>(response); +} diff --git a/client/src/api/common.test.ts b/client/src/api/common.test.ts new file mode 100644 index 0000000..141f094 --- /dev/null +++ b/client/src/api/common.test.ts @@ -0,0 +1,71 @@ +import { handleResponse } from './common'; + +function mockResponse(body: unknown, status = 200): Response { + return { + ok: status >= 200 && status < 300, + status, + json: () => Promise.resolve(body), + } as Response; +} + +describe('handleResponse', () => { + it('returns parsed JSON on success', async () => { + const data = { id: '1', name: 'Test' }; + const result = await handleResponse(mockResponse(data)); + expect(result).toEqual(data); + }); + + it('throws error with message from response body', async () => { + await expect( + handleResponse(mockResponse({ message: 'Not found' }, 404)), + ).rejects.toThrow('Not found'); + }); + + it('throws error with error field from response body', async () => { + await expect( + handleResponse(mockResponse({ error: 'Bad request' }, 400)), + ).rejects.toThrow('Bad request'); + }); + + it('throws generic error when response body has no message', async () => { + await expect( + handleResponse(mockResponse({}, 500)), + ).rejects.toThrow('HTTP error 500'); + }); + + it('throws fallback error when response body is not JSON', async () => { + const response = { + ok: false, + status: 500, + json: () => Promise.reject(new Error('not JSON')), + } as Response; + + await expect(handleResponse(response)).rejects.toThrow('Request failed'); + }); + + it('dispatches auth:expired event on 401 response', async () => { + const handler = jest.fn(); + window.addEventListener('auth:expired', handler); + + await expect( + handleResponse(mockResponse({ error: 'Not authenticated' }, 401)), + ).rejects.toThrow('Not authenticated'); + + expect(handler).toHaveBeenCalledTimes(1); + + window.removeEventListener('auth:expired', handler); + }); + + it('does not dispatch auth:expired event on non-401 errors', async () => { + const handler = jest.fn(); + window.addEventListener('auth:expired', handler); + + await expect( + handleResponse(mockResponse({ message: 'Server error' }, 500)), + ).rejects.toThrow('Server error'); + + expect(handler).not.toHaveBeenCalled(); + + window.removeEventListener('auth:expired', handler); + }); +}); diff --git a/client/src/api/common.ts b/client/src/api/common.ts index 19100e6..22a623a 100644 --- a/client/src/api/common.ts +++ b/client/src/api/common.ts @@ -7,6 +7,9 @@ */ export async function handleResponse(response: Response): Promise { if (!response.ok) { + if (response.status === 401) { + window.dispatchEvent(new Event('auth:expired')); + } const error = await response.json().catch(() => ({ message: 'Request failed' })); throw new Error(error.message || error.error || `HTTP error ${response.status}`); } diff --git a/client/src/api/dependencies.ts b/client/src/api/dependencies.ts index d9fd910..bcfed45 100644 --- a/client/src/api/dependencies.ts +++ b/client/src/api/dependencies.ts @@ -31,3 +31,31 @@ export async function clearDependencyOverrides(id: string): Promise { throw new Error(error.message || `HTTP error ${response.status}`); } } + +export interface DependencyEnrichmentInput { + displayName?: string | null; + description?: string | null; + impact?: string | null; +} + +export async function enrichDependency( + id: string, + input: DependencyEnrichmentInput, +): Promise { + const response = await fetch(`/api/dependencies/${id}/enrich`, { + method: 'PATCH', + headers: withCsrfToken({ 'Content-Type': 'application/json' }), + body: JSON.stringify(input), + credentials: 'include', + }); + return handleResponse(response); +} + +export async function listDiscoveredDependencies( + serviceId: string, +): Promise { + const response = await fetch(`/api/services/${serviceId}/discovered-dependencies`, { + credentials: 'include', + }); + return handleResponse(response); +} diff --git a/client/src/api/externalNodes.ts b/client/src/api/externalNodes.ts new file mode 100644 index 0000000..aeece98 --- /dev/null +++ b/client/src/api/externalNodes.ts @@ -0,0 +1,55 @@ +import { handleResponse } from './common'; +import { withCsrfToken } from './csrf'; + +export interface ExternalNodeEnrichment { + id: string; + canonical_name: string; + display_name: string | null; + description: string | null; + impact: string | null; + contact: string | null; + service_type: string | null; + created_at: string; + updated_at: string; + updated_by: string | null; +} + +export interface UpsertExternalNodeInput { + displayName?: string | null; + description?: string | null; + impact?: string | null; + contact?: Record | null; + serviceType?: string | null; +} + +export async function fetchExternalNodes(): Promise { + const response = await fetch('/api/external-nodes', { + credentials: 'include', + }); + return handleResponse(response); +} + +export async function upsertExternalNode( + canonicalName: string, + input: UpsertExternalNodeInput, +): Promise { + const response = await fetch(`/api/external-nodes/${encodeURIComponent(canonicalName)}`, { + method: 'PUT', + headers: withCsrfToken({ 'Content-Type': 'application/json' }), + body: JSON.stringify(input), + credentials: 'include', + }); + return handleResponse(response); +} + +export async function deleteExternalNode(canonicalName: string): Promise { + const response = await fetch(`/api/external-nodes/${encodeURIComponent(canonicalName)}`, { + method: 'DELETE', + headers: withCsrfToken(), + credentials: 'include', + }); + if (!response.ok) { + const error = await response.json().catch(() => ({ message: 'Delete failed' })); + throw new Error(error.message || `HTTP error ${response.status}`); + } +} diff --git a/client/src/api/otlpStats.ts b/client/src/api/otlpStats.ts new file mode 100644 index 0000000..70f4dc4 --- /dev/null +++ b/client/src/api/otlpStats.ts @@ -0,0 +1,76 @@ +import { handleResponse } from './common'; +import type { OtlpStatsResponse, AdminOtlpStatsResponse, ApiKeyUsageResponse, AdminOtlpUsageResponse } from '../types/otlpStats'; + +export async function getTeamOtlpStats(teamId: string): Promise { + const response = await fetch(`/api/teams/${teamId}/otlp-stats`, { + credentials: 'include', + }); + return handleResponse(response); +} + +export async function getAdminOtlpStats(): Promise { + const response = await fetch('/api/admin/otlp-stats', { + credentials: 'include', + }); + return handleResponse(response); +} + +export async function getApiKeyUsage( + teamId: string, + keyId: string, + params: { from: string; to: string; granularity: 'minute' | 'hour' }, +): Promise { + const qs = new URLSearchParams(params).toString(); + const response = await fetch(`/api/teams/${teamId}/api-keys/${keyId}/usage?${qs}`, { + credentials: 'include', + }); + return handleResponse(response); +} + +export async function updateApiKeyRateLimit( + teamId: string, + keyId: string, + rateLimit: number | null, +): Promise { + const response = await fetch(`/api/teams/${teamId}/api-keys/${keyId}/rate-limit`, { + method: 'PATCH', + credentials: 'include', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ rate_limit_rpm: rateLimit }), + }); + return handleResponse(response); +} + +export async function getAdminApiKeyUsage( + keyId: string, + params: { from: string; to: string; granularity: 'minute' | 'hour' }, +): Promise { + const qs = new URLSearchParams(params).toString(); + const response = await fetch(`/api/admin/api-keys/${keyId}/usage?${qs}`, { + credentials: 'include', + }); + return handleResponse(response); +} + +export async function updateAdminApiKeyRateLimit( + keyId: string, + payload: { rate_limit_rpm: number | null; admin_locked?: boolean }, +): Promise { + const response = await fetch(`/api/admin/api-keys/${keyId}/rate-limit`, { + method: 'PATCH', + credentials: 'include', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + }); + return handleResponse(response); +} + +export async function getAdminOtlpUsage( + params: { from: string; to: string }, +): Promise { + const qs = new URLSearchParams(params).toString(); + const response = await fetch(`/api/admin/otlp-usage?${qs}`, { + credentials: 'include', + }); + return handleResponse(response); +} diff --git a/client/src/components/Charts/ApiKeyUsageChart.module.css b/client/src/components/Charts/ApiKeyUsageChart.module.css new file mode 100644 index 0000000..8d9b0fe --- /dev/null +++ b/client/src/components/Charts/ApiKeyUsageChart.module.css @@ -0,0 +1,104 @@ +.container { + background: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: 8px; + overflow: hidden; +} + +.header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 12px 16px; + border-bottom: 1px solid var(--color-border); +} + +.title { + font-size: 13px; + font-weight: 600; + color: var(--color-text-heading); + margin: 0; +} + +.chartArea { + padding: 12px 8px 8px; + min-height: 240px; + display: flex; + align-items: center; + justify-content: center; +} + +.loadingState, +.errorState, +.emptyState { + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + gap: 8px; + padding: 40px 20px; + color: var(--color-text-muted); + font-size: 13px; + width: 100%; +} + +.spinner { + width: 20px; + height: 20px; + border: 2px solid var(--color-border); + border-top-color: var(--color-accent); + border-radius: 50%; + animation: spin 1s linear infinite; +} + +@keyframes spin { + to { + transform: rotate(360deg); + } +} + +.retryButton { + padding: 4px 12px; + border: 1px solid var(--color-border); + border-radius: 4px; + background: var(--color-bg-card); + color: var(--color-accent); + font-size: 12px; + cursor: pointer; + transition: all var(--duration-fast); +} + +.retryButton:hover { + background: var(--color-bg-hover); +} + +.tooltip { + background: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: 6px; + padding: 8px 12px; + box-shadow: var(--shadow-md); + font-size: 12px; +} + +.tooltipTime { + font-weight: 600; + color: var(--color-text-heading); + margin-bottom: 4px; +} + +.tooltipRow { + display: flex; + align-items: center; + gap: 6px; + color: var(--color-text-primary); + line-height: 1.6; +} + +.tooltipDot { + display: inline-block; + width: 8px; + height: 8px; + border-radius: 50%; + flex-shrink: 0; +} diff --git a/client/src/components/Charts/ApiKeyUsageChart.test.tsx b/client/src/components/Charts/ApiKeyUsageChart.test.tsx new file mode 100644 index 0000000..9efdcf2 --- /dev/null +++ b/client/src/components/Charts/ApiKeyUsageChart.test.tsx @@ -0,0 +1,404 @@ +import { render, screen, waitFor, fireEvent } from '@testing-library/react'; +import { ApiKeyUsageChart } from './ApiKeyUsageChart'; + +// Mock recharts to avoid jsdom SVG issues +jest.mock('recharts', () => { + const OriginalModule = jest.requireActual('recharts'); + return { + ...OriginalModule, + ResponsiveContainer: ({ children }: { children: React.ReactNode }) => ( +
{children}
+ ), + }; +}); + +const mockFetch = jest.fn(); +global.fetch = mockFetch; + +function jsonResponse(data: unknown, status = 200) { + return { + ok: status >= 200 && status < 300, + status, + json: () => Promise.resolve(data), + }; +} + +const mockBuckets = [ + { bucket_start: '2026-04-04T10:00:00', push_count: 120, rejected_count: 0 }, + { bucket_start: '2026-04-04T10:01:00', push_count: 95, rejected_count: 5 }, + { bucket_start: '2026-04-04T10:02:00', push_count: 110, rejected_count: 0 }, +]; + +const defaultUsageResponse = { + api_key_id: 'k1', + granularity: 'minute' as const, + from: '2026-04-03T10:00:00Z', + to: '2026-04-04T10:00:00Z', + buckets: mockBuckets, +}; + +const emptyUsageResponse = { + api_key_id: 'k1', + granularity: 'minute' as const, + from: '2026-04-03T10:00:00Z', + to: '2026-04-04T10:00:00Z', + buckets: [], +}; + +beforeEach(() => { + mockFetch.mockReset(); + localStorage.clear(); +}); + +describe('ApiKeyUsageChart', () => { + it('renders loading state initially', () => { + // Return a never-resolving promise to keep loading state + mockFetch.mockReturnValue(new Promise(() => {})); + + render( + + ); + + expect(screen.getByText('Loading usage data...')).toBeInTheDocument(); + }); + + it('renders chart content when data is returned', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Loading usage data...')).not.toBeInTheDocument(); + expect(screen.queryByText('No push data for this period.')).not.toBeInTheDocument(); + }); + + it('renders empty state when buckets is empty', async () => { + mockFetch.mockResolvedValue(jsonResponse(emptyUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByText('No push data for this period.')).toBeInTheDocument(); + }); + + expect(screen.queryByTestId('responsive-container')).not.toBeInTheDocument(); + }); + + it('renders error state when fetch fails', async () => { + mockFetch.mockResolvedValue(jsonResponse({ message: 'Server error' }, 500)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByText('Server error')).toBeInTheDocument(); + }); + + expect(screen.getByText('Retry')).toBeInTheDocument(); + }); + + it('renders title with key name and prefix', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + expect(screen.getByText('Prod Key (dps_abc123) — Usage')).toBeInTheDocument(); + }); + + it('renders title with prefix only when keyName is empty', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + expect(screen.getByText('dps_abc123 — Usage')).toBeInTheDocument(); + }); + + it('uses team endpoint when teamId is provided and isAdmin is false', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('/api/teams/t1/api-keys/k1/usage?'), + expect.objectContaining({ credentials: 'include' }) + ); + }); + + it('uses admin endpoint when isAdmin is true', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('/api/admin/api-keys/k1/usage?'), + expect.objectContaining({ credentials: 'include' }) + ); + }); + + it('uses admin endpoint when teamId is not provided', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('/api/admin/api-keys/k1/usage?'), + expect.objectContaining({ credentials: 'include' }) + ); + }); + + it('default range uses granularity=minute', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + // Default range is 24h which uses minute granularity + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('granularity=minute'), + expect.anything() + ); + }); + + it('switching to 7d range uses granularity=hour', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + mockFetch.mockClear(); + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + // Click the 7d range button + fireEvent.click(screen.getByText('7d')); + + await waitFor(() => { + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('granularity=hour'), + expect.anything() + ); + }); + }); + + it('switching to 30d range uses granularity=hour', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + mockFetch.mockClear(); + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + fireEvent.click(screen.getByText('30d')); + + await waitFor(() => { + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('granularity=hour'), + expect.anything() + ); + }); + }); + + it('switching to 1h range uses granularity=minute', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + mockFetch.mockClear(); + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + fireEvent.click(screen.getByText('1h')); + + await waitFor(() => { + expect(mockFetch).toHaveBeenCalledWith( + expect.stringContaining('granularity=minute'), + expect.anything() + ); + }); + }); + + it('switching time range triggers a new fetch', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + const initialCallCount = mockFetch.mock.calls.length; + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + fireEvent.click(screen.getByText('6h')); + + await waitFor(() => { + expect(mockFetch.mock.calls.length).toBeGreaterThan(initialCallCount); + }); + }); + + it('retry button re-fetches data after error', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse({ message: 'Timeout' }, 500)); + + render( + + ); + + await waitFor(() => { + expect(screen.getByText('Timeout')).toBeInTheDocument(); + }); + + mockFetch.mockResolvedValueOnce(jsonResponse(defaultUsageResponse)); + fireEvent.click(screen.getByText('Retry')); + + await waitFor(() => { + expect(screen.getByTestId('responsive-container')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Timeout')).not.toBeInTheDocument(); + }); + + it('renders all time range buttons', async () => { + mockFetch.mockResolvedValue(jsonResponse(defaultUsageResponse)); + + render( + + ); + + expect(screen.getByText('1h')).toBeInTheDocument(); + expect(screen.getByText('6h')).toBeInTheDocument(); + expect(screen.getByText('24h')).toBeInTheDocument(); + expect(screen.getByText('7d')).toBeInTheDocument(); + expect(screen.getByText('30d')).toBeInTheDocument(); + }); +}); diff --git a/client/src/components/Charts/ApiKeyUsageChart.tsx b/client/src/components/Charts/ApiKeyUsageChart.tsx new file mode 100644 index 0000000..89658ce --- /dev/null +++ b/client/src/components/Charts/ApiKeyUsageChart.tsx @@ -0,0 +1,199 @@ +import { useCallback, useEffect, useState } from 'react'; +import { + Bar, + CartesianGrid, + ComposedChart, + ResponsiveContainer, + Tooltip, + XAxis, + YAxis, +} from 'recharts'; +import { getApiKeyUsage, getAdminApiKeyUsage } from '../../api/otlpStats'; +import { ApiKeyUsageBucket } from '../../types/otlpStats'; +import { LatencyRange } from '../../types/chart'; +import { parseUtcDate } from '../../utils/formatting'; +import { TimeRangeSelector } from './TimeRangeSelector'; +import styles from './ApiKeyUsageChart.module.css'; + +interface ApiKeyUsageChartProps { + teamId?: string; + apiKeyId: string; + keyName: string; + keyPrefix: string; + isAdmin?: boolean; +} + +type UsageRange = '1h' | '6h' | '24h' | '7d' | '30d'; +const USAGE_RANGES: UsageRange[] = ['1h', '6h', '24h', '7d', '30d']; + +const RANGE_DURATIONS_MS: Record = { + '1h': 60 * 60 * 1000, + '6h': 6 * 60 * 60 * 1000, + '24h': 24 * 60 * 60 * 1000, + '7d': 7 * 24 * 60 * 60 * 1000, + '30d': 30 * 24 * 60 * 60 * 1000, +}; + +function getGranularity(range: UsageRange): 'minute' | 'hour' { + return range === '7d' || range === '30d' ? 'hour' : 'minute'; +} + +interface ChartDataPoint extends ApiKeyUsageBucket { + label: string; +} + +function formatBucketTime(bucketStart: string, granularity: 'minute' | 'hour'): string { + const date = parseUtcDate(bucketStart); + if (granularity === 'minute') { + return date.toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit' }); + } + return date.toLocaleDateString(undefined, { month: 'short', day: 'numeric', hour: '2-digit' }); +} + +function formatTooltipTimestamp(bucketStart: string): string { + const date = parseUtcDate(bucketStart); + return date.toLocaleString(undefined, { + month: 'short', + day: 'numeric', + hour: '2-digit', + minute: '2-digit', + }); +} + +export function ApiKeyUsageChart({ + teamId, + apiKeyId, + keyName, + keyPrefix, + isAdmin, +}: ApiKeyUsageChartProps) { + const [data, setData] = useState([]); + const [range, setRange] = useState('24h'); + const [isLoading, setIsLoading] = useState(true); + const [error, setError] = useState(null); + + const loadData = useCallback( + async (selectedRange: UsageRange) => { + setIsLoading(true); + setError(null); + try { + const now = new Date(); + const from = new Date(now.getTime() - RANGE_DURATIONS_MS[selectedRange]).toISOString(); // eslint-disable-line security/detect-object-injection + const to = now.toISOString(); + const granularity = getGranularity(selectedRange); + + const response = + isAdmin || !teamId + ? await getAdminApiKeyUsage(apiKeyId, { from, to, granularity }) + : await getApiKeyUsage(teamId, apiKeyId, { from, to, granularity }); + + setData( + response.buckets.map((b: ApiKeyUsageBucket) => ({ + ...b, + label: formatBucketTime(b.bucket_start, granularity), + })) + ); + } catch (err) { + setError(err instanceof Error ? err.message : 'Failed to load usage data'); + } finally { + setIsLoading(false); + } + }, + [apiKeyId, teamId, isAdmin] + ); + + useEffect(() => { + loadData(range); + }, [range, loadData]); + + const handleRangeChange = useCallback((newRange: LatencyRange | string) => { + setRange(newRange as UsageRange); + }, []); + + const renderTooltipContent = useCallback( + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (props: any) => { + const { payload, active } = props; + if (!active || !payload || payload.length === 0) return null; + const point = payload[0].payload as ChartDataPoint; + return ( +
+
{formatTooltipTimestamp(point.bucket_start)}
+
+ + Pushes: {point.push_count.toLocaleString()} +
+ {point.rejected_count > 0 && ( +
+ + Rejected: {point.rejected_count.toLocaleString()} +
+ )} +
+ ); + }, + [] + ); + + const title = keyName ? `${keyName} (${keyPrefix}) — Usage` : `${keyPrefix} — Usage`; + + return ( +
+
+

{title}

+ +
+
+ {isLoading ? ( +
+
+ Loading usage data... +
+ ) : error ? ( +
+ {error} + +
+ ) : data.length === 0 ? ( +
+ No push data for this period. +
+ ) : ( + + + + + value.toLocaleString()} + width={60} + /> + + + + + + )} +
+
+ ); +} diff --git a/client/src/components/Charts/index.ts b/client/src/components/Charts/index.ts index 50b1c0f..6e27f0c 100644 --- a/client/src/components/Charts/index.ts +++ b/client/src/components/Charts/index.ts @@ -1,3 +1,4 @@ +export { ApiKeyUsageChart } from './ApiKeyUsageChart'; export { LatencyChart } from './LatencyChart'; export { HealthTimeline } from './HealthTimeline'; export { TimeRangeSelector } from './TimeRangeSelector'; diff --git a/client/src/components/Layout/Layout.tsx b/client/src/components/Layout/Layout.tsx index 23bf3b6..e066393 100644 --- a/client/src/components/Layout/Layout.tsx +++ b/client/src/components/Layout/Layout.tsx @@ -219,6 +219,19 @@ function Layout() { Manifests + + `${styles.navLink} ${isActive ? styles.navLinkActive : ''}` + } + onClick={closeSidebar} + title="OTLP" + > + + + + OTLP + )} diff --git a/client/src/components/Login/Login.module.css b/client/src/components/Login/Login.module.css index a0642f0..617ff11 100644 --- a/client/src/components/Login/Login.module.css +++ b/client/src/components/Login/Login.module.css @@ -17,12 +17,22 @@ max-width: 400px; } -.title { - font-size: var(--font-2xl); - font-weight: var(--font-semibold); - color: var(--color-text); - text-align: center; - margin: 0 0 var(--space-1) 0; +.logoContainer { + display: flex; + align-items: center; + justify-content: center; + gap: var(--space-2); + margin-bottom: var(--space-1); +} + +.logoIcon { + width: 40px; + height: 40px; +} + +.logoTitle { + height: 28px; + width: auto; } .subtitle { diff --git a/client/src/components/Login/Login.tsx b/client/src/components/Login/Login.tsx index ab05989..4e421fd 100644 --- a/client/src/components/Login/Login.tsx +++ b/client/src/components/Login/Login.tsx @@ -72,7 +72,10 @@ function Login() { return (
-

Depsera

+
+ + Depsera +

Sign in to continue

{displayError && ( diff --git a/client/src/components/common/Modal.tsx b/client/src/components/common/Modal.tsx index 8ad32d5..7673d76 100644 --- a/client/src/components/common/Modal.tsx +++ b/client/src/components/common/Modal.tsx @@ -54,7 +54,7 @@ function Modal({ isOpen, onClose, title, children, size = 'md' }: ModalProps) { return ( diff --git a/client/src/components/pages/Admin/OtlpAdmin.module.css b/client/src/components/pages/Admin/OtlpAdmin.module.css new file mode 100644 index 0000000..1222ae0 --- /dev/null +++ b/client/src/components/pages/Admin/OtlpAdmin.module.css @@ -0,0 +1,588 @@ +.container { + display: flex; + flex-direction: column; + gap: var(--space-4); + padding: var(--space-4); + height: 100%; + overflow-y: auto; +} + +.header { + display: flex; + justify-content: space-between; + align-items: center; +} + +.headerTitle { + font-size: var(--font-xl); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +.summaryGrid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(10rem, 1fr)); + gap: var(--space-3); +} + +.summaryCard { + padding: var(--space-3); + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + text-align: center; +} + +.summaryValue { + font-size: var(--font-2xl); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +.summaryLabel { + font-size: var(--font-xs); + color: var(--color-text-muted); + margin: 0.25rem 0 0; +} + +.summaryCardWarning { + composes: summaryCard; + border-color: var(--color-warning-border, rgba(234, 179, 8, 0.3)); + background-color: var(--color-warning-bg, rgba(234, 179, 8, 0.06)); +} + +.summaryCardError { + composes: summaryCard; + border-color: var(--color-critical-border, rgba(239, 68, 68, 0.3)); + background-color: var(--color-critical-bg, rgba(239, 68, 68, 0.06)); +} + +.teamSection { + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + overflow: hidden; +} + +.teamHeader { + display: flex; + align-items: center; + gap: var(--space-2); + padding: 0.75rem 1rem; + background-color: var(--color-surface); + border-bottom: 1px solid var(--color-border); + cursor: pointer; + user-select: none; +} + +.teamHeader:hover { + background-color: var(--color-bg-hover); +} + +.teamName { + font-size: var(--font-base); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; + flex: 1; +} + +.teamMeta { + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.chevron { + color: var(--color-text-muted); + transition: transform var(--duration-fast); +} + +.chevronOpen { + composes: chevron; + transform: rotate(90deg); +} + +.teamBody { + padding: var(--space-3); + display: flex; + flex-direction: column; + gap: var(--space-3); +} + +.tableWrapper { + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + overflow-x: auto; +} + +.table { + width: 100%; + border-collapse: collapse; + font-size: var(--font-sm); +} + +.table th { + text-align: left; + padding: 0.5rem 0.75rem; + font-weight: 500; + color: var(--color-text-muted); + border-bottom: 1px solid var(--color-border); + background-color: var(--color-surface); + white-space: nowrap; +} + +.table td { + padding: 0.5rem 0.75rem; + color: var(--color-text); + border-bottom: 1px solid var(--color-border); +} + +.table tbody tr:last-child td { + border-bottom: none; +} + +.table tbody tr:hover { + background-color: var(--color-bg-hover); +} + +.badge { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: var(--font-xs); + font-weight: 500; + border-radius: 9999px; + white-space: nowrap; +} + +.badgeSuccess { + composes: badge; + color: var(--color-healthy); + background-color: var(--color-success-bg, rgba(34, 197, 94, 0.08)); +} + +.badgeError { + composes: badge; + color: var(--color-critical); + background-color: var(--color-critical-bg, rgba(239, 68, 68, 0.06)); +} + +.badgeNeutral { + composes: badge; + color: var(--color-text-muted); + background-color: var(--color-surface); +} + +.badgeInactive { + composes: badge; + color: var(--color-text-muted); + background-color: var(--color-bg-hover); +} + +.loading { + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + gap: var(--space-3); + padding: var(--space-8); + color: var(--color-text-muted); +} + +.spinner { + width: 1.5rem; + height: 1.5rem; + border: 2px solid var(--color-border); + border-top-color: var(--color-accent); + border-radius: 50%; + animation: spin 0.6s linear infinite; +} + +@keyframes spin { + to { transform: rotate(360deg); } +} + +.error { + padding: 0.75rem 1rem; + color: var(--color-critical); + background-color: var(--color-critical-bg, rgba(239, 68, 68, 0.06)); + border: 1px solid var(--color-critical-border, rgba(239, 68, 68, 0.3)); + border-radius: var(--radius-md); +} + +.retryButton { + margin-top: 0.5rem; + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); + background-color: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + cursor: pointer; +} + +.retryButton:hover { + background-color: var(--color-bg-hover); +} + +.emptyState { + display: flex; + flex-direction: column; + align-items: center; + padding: var(--space-8); + color: var(--color-text-muted); + text-align: center; +} + +.emptyState p { + margin: 0.5rem 0 0; +} + +.emptyIcon { + opacity: 0.4; +} + +.errorText { + font-size: var(--font-xs); + color: var(--color-critical); + max-width: 20rem; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.errorText:hover { + white-space: normal; + overflow: visible; +} + +.keyList { + display: flex; + flex-wrap: wrap; + gap: 0.375rem; +} + +.keyChip { + display: inline-flex; + align-items: center; + gap: 0.375rem; + padding: 0.25rem 0.5rem; + font-size: var(--font-xs); + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + color: var(--color-text-muted); +} + +.keyChipName { + color: var(--color-text); + font-weight: 500; +} + +/* Usage overview section */ +.usageOverviewSection { + display: flex; + flex-direction: column; + gap: var(--space-3); + padding: var(--space-3); + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); +} + +.sectionTitle { + font-size: var(--font-base); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +/* Key card styles */ +.keyList { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.keyCard { + display: flex; + flex-direction: column; + gap: 0.5rem; + padding: 0.75rem 1rem; + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + font-size: var(--font-sm); +} + +.keyCardAmber { + composes: keyCard; + border-color: var(--color-warning-border, rgba(234, 179, 8, 0.3)); + background-color: var(--color-warning-bg, rgba(234, 179, 8, 0.06)); +} + +.keyCardRed { + composes: keyCard; + border-color: var(--color-critical-border, rgba(239, 68, 68, 0.3)); + background-color: var(--color-critical-bg, rgba(239, 68, 68, 0.06)); +} + +.keyCardHeader { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: var(--space-3); +} + +.keyCardInfo { + display: flex; + flex-direction: column; + gap: 0.125rem; + min-width: 0; +} + +.keyName { + display: flex; + align-items: center; + gap: 0.5rem; + font-weight: 500; + color: var(--color-text); +} + +.keyPrefix { + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.keyMeta { + font-size: var(--font-xs); + color: var(--color-text-muted); + text-align: right; + flex-shrink: 0; +} + +/* Warning badges */ +.badgeWarning { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: var(--font-xs); + font-weight: 500; + border-radius: 9999px; + white-space: nowrap; + color: var(--color-warning, #d97706); + background-color: var(--color-warning-bg, rgba(234, 179, 8, 0.08)); +} + +.badgeMutedWarning { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: var(--font-xs); + font-weight: 500; + border-radius: 9999px; + white-space: nowrap; + color: var(--color-text-muted); + background-color: var(--color-bg-hover); +} + +/* Usage summary row */ +.usageSummary { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 0.5rem; + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.rejectedWarning { + color: var(--color-warning, #d97706); + font-weight: 500; +} + +/* Rate limit row */ +.rateLimitRow { + display: flex; + align-items: center; + gap: 0.5rem; + font-size: var(--font-xs); +} + +.rateLimitText { + color: var(--color-text-muted); +} + +.rateLimitSuffix { + color: var(--color-text-muted); + opacity: 0.7; +} + +.lockIcon { + color: var(--color-text-muted); + flex-shrink: 0; +} + +.lockIndicator { + display: inline-flex; + align-items: center; + gap: 0.25rem; + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.editButton { + display: inline-flex; + align-items: center; + padding: 0.25rem; + color: var(--color-text-muted); + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast), border-color var(--duration-fast); + flex-shrink: 0; +} + +.editButton:hover { + color: var(--color-accent); + border-color: var(--color-accent); +} + +.expandButton { + display: inline-flex; + align-items: center; + gap: 0.25rem; + margin-left: auto; + padding: 0.25rem 0.5rem; + font-size: var(--font-xs); + color: var(--color-text-muted); + background: none; + border: 1px solid var(--color-border); + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast), border-color var(--duration-fast); +} + +.expandButton:hover { + color: var(--color-accent); + border-color: var(--color-accent); +} + +/* Chart container */ +.chartContainer { + border-top: 1px solid var(--color-border); + padding-top: 0.5rem; +} + +/* Rate limit edit dialog */ +.rateLimitDialog { + display: flex; + flex-direction: column; + gap: 0.75rem; +} + +.rateLimitDialogDesc { + font-size: var(--font-sm); + color: var(--color-text-secondary); + margin: 0; + line-height: 1.5; +} + +.rateLimitDialogLabel { + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); +} + +.rateLimitDialogInput { + padding: 0.5rem 0.75rem; + font-size: var(--font-sm); + border: 1px solid var(--color-border-input); + border-radius: var(--radius-md); + background-color: var(--color-bg-input); + color: var(--color-text); +} + +.rateLimitDialogInput:focus { + outline: none; + border-color: var(--color-accent); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +.rateLimitDialogError { + font-size: var(--font-xs); + color: var(--color-critical); + margin: -0.25rem 0 0; +} + +.rateLimitDialogActions { + display: flex; + justify-content: space-between; + align-items: center; + margin-top: 0.25rem; +} + +.rateLimitDialogRight { + display: flex; + gap: 0.5rem; +} + +.dialogSecondaryButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); + background-color: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + cursor: pointer; +} + +.dialogSecondaryButton:hover:not(:disabled) { + background-color: var(--color-bg-hover); +} + +.dialogSecondaryButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.dialogPrimaryButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text-inverse); + background-color: var(--color-accent); + border: 1px solid transparent; + border-radius: var(--radius-md); + cursor: pointer; +} + +.dialogPrimaryButton:hover:not(:disabled) { + background-color: var(--color-accent-hover); +} + +.dialogPrimaryButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +/* Lock checkbox */ +.lockCheckboxLabel { + display: flex; + align-items: center; + gap: 0.5rem; + font-size: var(--font-sm); + color: var(--color-text); + cursor: pointer; +} + +.lockNote { + font-size: var(--font-xs); + color: var(--color-text-muted); + margin: -0.25rem 0 0; + padding-left: 1.5rem; +} diff --git a/client/src/components/pages/Admin/OtlpAdmin.test.tsx b/client/src/components/pages/Admin/OtlpAdmin.test.tsx new file mode 100644 index 0000000..27aade4 --- /dev/null +++ b/client/src/components/pages/Admin/OtlpAdmin.test.tsx @@ -0,0 +1,281 @@ +import { render, screen, fireEvent, waitFor } from '@testing-library/react'; +import OtlpAdmin from './OtlpAdmin'; + +beforeAll(() => { + HTMLDialogElement.prototype.showModal = jest.fn(); + HTMLDialogElement.prototype.close = jest.fn(); +}); + +const mockFetch = jest.fn(); +global.fetch = mockFetch; + +function jsonResponse(data: unknown, status = 200) { + return { + ok: status >= 200 && status < 300, + status, + json: () => Promise.resolve(data), + }; +} + +const baseKey = { + id: 'k1', + name: 'Prod Key', + key_prefix: 'dps_abc123', + last_used_at: '2026-04-04T10:00:00Z', + created_at: '2026-03-01T10:00:00Z', + rate_limit_rpm: 150000, + rate_limit_is_custom: false, + rate_limit_admin_locked: false, + usage_1h: 500, + usage_24h: 8000, + usage_7d: 50000, + rejected_24h: 0, + rejected_7d: 0, +}; + +const baseTeam = { + team_id: 't1', + team_name: 'Alpha Team', + services: [ + { + id: 's1', + name: 'my-service', + is_active: 1, + last_push_success: 1, + last_push_error: null, + last_push_warnings: null, + last_push_at: '2026-04-04T09:00:00Z', + dependency_count: 3, + errors_24h: 0, + schema_config: null, + }, + ], + apiKeys: [baseKey], +}; + +function makeAdminStatsResponse(keyOverrides = {}) { + return { + teams: [{ + ...baseTeam, + apiKeys: [{ ...baseKey, ...keyOverrides }], + }], + summary: { + total_otlp_services: 1, + active_services: 1, + services_with_errors: 0, + services_never_pushed: 0, + total_teams: 1, + }, + }; +} + +const emptyUsageResponse = { + from: '2026-03-28T00:00:00Z', + to: '2026-04-04T00:00:00Z', + buckets: [], +}; + +beforeEach(() => { + mockFetch.mockReset(); + localStorage.clear(); +}); + +function renderAdminWithData(keyOverrides = {}) { + // OtlpAdmin makes two parallel fetches: getAdminOtlpStats and getAdminOtlpUsage + mockFetch.mockImplementation((url: string) => { + if (url.includes('/api/admin/otlp-stats')) { + return Promise.resolve(jsonResponse(makeAdminStatsResponse(keyOverrides))); + } + if (url.includes('/api/admin/otlp-usage')) { + return Promise.resolve(jsonResponse(emptyUsageResponse)); + } + return Promise.resolve(jsonResponse({})); + }); + return render(); +} + +describe('OtlpAdmin', () => { + // --- Admin lock checkbox tests (DPS-102c) --- + + it('Lock checkbox is present in admin rate limit dialog', async () => { + renderAdminWithData(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + // Admin always has an edit button + fireEvent.click(screen.getByTitle('Edit rate limit')); + + expect(screen.getByText('Edit Rate Limit (Admin)')).toBeInTheDocument(); + expect(screen.getByLabelText(/Lock — prevent team from changing this limit/)).toBeInTheDocument(); + }); + + it('saving with lock checkbox checked sends admin_locked: true', async () => { + renderAdminWithData(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + // Check the lock checkbox + const lockCheckbox = screen.getByLabelText(/Lock — prevent team from changing this limit/); + fireEvent.click(lockCheckbox); + + // Lock note should appear + expect(screen.getByText('Team members will see this limit but cannot change it.')).toBeInTheDocument(); + + // Enter a value + const input = screen.getByPlaceholderText('150000'); + fireEvent.change(input, { target: { value: '100000' } }); + + // Mock the PATCH and reload + mockFetch.mockImplementation((url: string, opts?: RequestInit) => { + if (opts?.method === 'PATCH') { + return Promise.resolve(jsonResponse({ ok: true })); + } + if (url.includes('/api/admin/otlp-stats')) { + return Promise.resolve(jsonResponse(makeAdminStatsResponse())); + } + if (url.includes('/api/admin/otlp-usage')) { + return Promise.resolve(jsonResponse(emptyUsageResponse)); + } + return Promise.resolve(jsonResponse({})); + }); + + fireEvent.click(screen.getByText('Save')); + + await waitFor(() => { + const patchCall = mockFetch.mock.calls.find( + (c: [string, RequestInit?]) => c[1]?.method === 'PATCH' + ); + expect(patchCall).toBeDefined(); + const body = JSON.parse(patchCall![1]!.body as string); + expect(body.admin_locked).toBe(true); + expect(body.rate_limit_rpm).toBe(100000); + }); + }); + + it('saving with lock checkbox unchecked sends admin_locked: false', async () => { + renderAdminWithData({ rate_limit_admin_locked: true }); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + // Uncheck the lock checkbox (it should be checked by default since key is locked) + const lockCheckbox = screen.getByLabelText(/Lock — prevent team from changing this limit/) as HTMLInputElement; + expect(lockCheckbox.checked).toBe(true); + fireEvent.click(lockCheckbox); + + // Mock the PATCH and reload + mockFetch.mockImplementation((url: string, opts?: RequestInit) => { + if (opts?.method === 'PATCH') { + return Promise.resolve(jsonResponse({ ok: true })); + } + if (url.includes('/api/admin/otlp-stats')) { + return Promise.resolve(jsonResponse(makeAdminStatsResponse())); + } + if (url.includes('/api/admin/otlp-usage')) { + return Promise.resolve(jsonResponse(emptyUsageResponse)); + } + return Promise.resolve(jsonResponse({})); + }); + + fireEvent.click(screen.getByText('Save')); + + await waitFor(() => { + const patchCall = mockFetch.mock.calls.find( + (c: [string, RequestInit?]) => c[1]?.method === 'PATCH' + ); + expect(patchCall).toBeDefined(); + const body = JSON.parse(patchCall![1]!.body as string); + expect(body.admin_locked).toBe(false); + }); + }); + + it('admin dialog allows input of 0 without validation error', async () => { + renderAdminWithData(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + const input = screen.getByPlaceholderText('150000'); + fireEvent.change(input, { target: { value: '0' } }); + + // Save should be enabled (0 is valid for admin) + expect(screen.getByText('Save')).not.toBeDisabled(); + }); + + it('admin dialog shows description mentioning "Enter 0 for unlimited"', async () => { + renderAdminWithData(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + expect(screen.getByText(/Enter 0 for unlimited/)).toBeInTheDocument(); + }); + + // --- Admin locked key display (DPS-102c) --- + + it('locked key shows admin lock indicator', async () => { + renderAdminWithData({ rate_limit_admin_locked: true }); + + await waitFor(() => { + expect(screen.getByText('Admin locked')).toBeInTheDocument(); + }); + + expect(screen.getByTitle('Locked by admin')).toBeInTheDocument(); + }); + + // --- Warning badge in admin view --- + + it('renders "Rate limited" badge when rejected_24h > 0', async () => { + renderAdminWithData({ rejected_24h: 42 }); + + await waitFor(() => { + expect(screen.getByText('Rate limited')).toBeInTheDocument(); + }); + }); + + it('renders muted 7d rejection text when rejected_7d > 0 but rejected_24h is 0', async () => { + renderAdminWithData({ rejected_24h: 0, rejected_7d: 15 }); + + await waitFor(() => { + expect(screen.getByText('15 rejected in 7d')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Rate limited')).not.toBeInTheDocument(); + }); + + it('does not render warning badge when no rejections', async () => { + renderAdminWithData({ rejected_24h: 0, rejected_7d: 0 }); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Rate limited')).not.toBeInTheDocument(); + expect(screen.queryByText(/rejected in 7d/)).not.toBeInTheDocument(); + }); + + // --- Admin edit button always shows --- + + it('admin always has edit button regardless of lock state', async () => { + renderAdminWithData({ rate_limit_admin_locked: true }); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + }); +}); diff --git a/client/src/components/pages/Admin/OtlpAdmin.tsx b/client/src/components/pages/Admin/OtlpAdmin.tsx new file mode 100644 index 0000000..ef8ea74 --- /dev/null +++ b/client/src/components/pages/Admin/OtlpAdmin.tsx @@ -0,0 +1,617 @@ +import { useState, useEffect, useCallback } from 'react'; +import { Activity, ChevronRight, ChevronDown, ChevronUp, Lock, Pencil } from 'lucide-react'; +import { getAdminOtlpStats, getAdminOtlpUsage, updateAdminApiKeyRateLimit } from '../../../api/otlpStats'; +import type { + AdminOtlpStatsResponse, + AdminOtlpTeamStats, + AdminOtlpUsageResponse, + OtlpApiKeyStats, +} from '../../../types/otlpStats'; +import { ApiKeyUsageChart } from '../../Charts'; +import Modal from '../../common/Modal'; +import { formatRelativeTime } from '../../../utils/formatting'; +import styles from './OtlpAdmin.module.css'; + +const DEFAULT_RATE_LIMIT_RPM = 150_000; + +function OtlpAdmin() { + const [data, setData] = useState(null); + const [usageData, setUsageData] = useState(null); + const [isLoading, setIsLoading] = useState(true); + const [error, setError] = useState(null); + const [expandedTeams, setExpandedTeams] = useState>(new Set()); + const [expandedCharts, setExpandedCharts] = useState>(new Set()); + + // Rate limit edit dialog state + const [editRateLimitKey, setEditRateLimitKey] = useState(null); + const [rateLimitInput, setRateLimitInput] = useState(''); + const [rateLimitError, setRateLimitError] = useState(null); + const [isSavingRateLimit, setIsSavingRateLimit] = useState(false); + const [adminLockChecked, setAdminLockChecked] = useState(false); + + const loadStats = useCallback(async () => { + try { + setIsLoading(true); + const now = new Date(); + const from = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000).toISOString(); + const to = now.toISOString(); + + const [statsResult, usageResult] = await Promise.all([ + getAdminOtlpStats(), + getAdminOtlpUsage({ from, to }), + ]); + + setData(statsResult); + setUsageData(usageResult); + setError(null); + // Expand all teams by default + setExpandedTeams(new Set(statsResult.teams.map(t => t.team_id))); + } catch (err) { + setError(err instanceof Error ? err.message : 'Failed to load OTLP stats'); + } finally { + setIsLoading(false); + } + }, []); + + useEffect(() => { + loadStats(); + }, [loadStats]); + + const toggleTeam = (teamId: string) => { + setExpandedTeams(prev => { + const next = new Set(prev); + if (next.has(teamId)) next.delete(teamId); + else next.add(teamId); + return next; + }); + }; + + const toggleChart = useCallback((keyId: string) => { + setExpandedCharts(prev => { + const next = new Set(prev); + if (next.has(keyId)) next.delete(keyId); + else next.add(keyId); + return next; + }); + }, []); + + const openRateLimitDialog = (key: OtlpApiKeyStats) => { + setEditRateLimitKey(key); + setRateLimitInput(key.rate_limit_is_custom ? String(key.rate_limit_rpm) : ''); + setAdminLockChecked(key.rate_limit_admin_locked); + setRateLimitError(null); + }; + + const closeRateLimitDialog = () => { + setEditRateLimitKey(null); + setRateLimitInput(''); + setRateLimitError(null); + setAdminLockChecked(false); + }; + + const validateRateLimitInput = (value: string): string | null => { + if (value === '') return null; // reset to default + const num = Number(value); + if (num === 0) return null; // admin can set unlimited + if (!Number.isInteger(num) || num < 0) return 'Must be a non-negative integer'; + return null; + }; + + const handleSaveRateLimit = async () => { + if (!editRateLimitKey) return; + const trimmed = rateLimitInput.trim(); + const validationError = validateRateLimitInput(trimmed); + if (validationError) { + setRateLimitError(validationError); + return; + } + const newLimit = trimmed === '' ? null : Number(trimmed); + try { + setIsSavingRateLimit(true); + await updateAdminApiKeyRateLimit(editRateLimitKey.id, { + rate_limit_rpm: newLimit, + admin_locked: adminLockChecked, + }); + closeRateLimitDialog(); + await loadStats(); + } catch (err) { + setRateLimitError(err instanceof Error ? err.message : 'Failed to update rate limit'); + } finally { + setIsSavingRateLimit(false); + } + }; + + const handleResetToDefault = async () => { + if (!editRateLimitKey) return; + try { + setIsSavingRateLimit(true); + await updateAdminApiKeyRateLimit(editRateLimitKey.id, { + rate_limit_rpm: null, + admin_locked: adminLockChecked, + }); + closeRateLimitDialog(); + await loadStats(); + } catch (err) { + setRateLimitError(err instanceof Error ? err.message : 'Failed to reset rate limit'); + } finally { + setIsSavingRateLimit(false); + } + }; + + if (isLoading) { + return ( +
+
+
+ Loading OTLP stats... +
+
+ ); + } + + if (error) { + return ( +
+
+

{error}

+ +
+
+ ); + } + + if (!data) return null; + + const { teams, summary } = data; + + // Derive usage overview from usageData + const usageOverview = deriveUsageOverview(usageData, teams); + + return ( +
+
+

OTLP Push Overview

+
+ + {/* Usage Overview */} + {usageOverview && ( +
+
+
+
{usageOverview.pushes24h.toLocaleString()}
+
Pushes (24h)
+
+
+
{usageOverview.pushes7d.toLocaleString()}
+
Pushes (7d)
+
+ {usageOverview.rejected24h > 0 && ( +
+
{usageOverview.rejected24h.toLocaleString()}
+
Rejected (24h)
+
+ )} + {usageOverview.rejected7d > 0 && ( +
0 ? styles.summaryCardError : styles.summaryCardWarning}> +
{usageOverview.rejected7d.toLocaleString()}
+
Rejected (7d)
+
+ )} +
+ + {usageOverview.topKeys.length > 0 && ( + <> +

Top 5 Keys by 7-day Volume

+
+ + + + + + + + + + {usageOverview.topKeys.map(k => ( + + + + + + ))} + +
KeyTeam7-day Pushes
{k.keyName}{k.teamName}{k.pushCount.toLocaleString()}
+
+ + )} +
+ )} + + {/* Global Summary */} +
+
+
{summary.total_otlp_services}
+
Total OTLP Services
+
+
+
{summary.active_services}
+
Active
+
+
0 ? styles.summaryCardError : styles.summaryCard}> +
{summary.services_with_errors}
+
With Errors
+
+
0 ? styles.summaryCardWarning : styles.summaryCard}> +
{summary.services_never_pushed}
+
Never Pushed
+
+
+
{summary.total_teams}
+
Teams
+
+
+ + {/* Per-team sections */} + {teams.length === 0 ? ( +
+ +

No OTLP services configured across any team.

+
+ ) : ( + teams.map(team => ( + toggleTeam(team.team_id)} + expandedCharts={expandedCharts} + onToggleChart={toggleChart} + onEditRateLimit={openRateLimitDialog} + /> + )) + )} + + {/* Admin Rate Limit Edit Dialog */} + + {editRateLimitKey && ( +
+

+ Set the rate limit for {editRateLimitKey.name} ({editRateLimitKey.key_prefix}...). + Leave empty to use the system default ({DEFAULT_RATE_LIMIT_RPM.toLocaleString()} req/min). + Enter 0 for unlimited. +

+ + { + setRateLimitInput(e.target.value); + setRateLimitError(null); + }} + placeholder={String(DEFAULT_RATE_LIMIT_RPM)} + className={styles.rateLimitDialogInput} + min={0} + disabled={isSavingRateLimit} + onKeyDown={(e) => { + if (e.key === 'Enter') handleSaveRateLimit(); + }} + /> + + {adminLockChecked && ( +

+ Team members will see this limit but cannot change it. +

+ )} + {rateLimitError && ( +

{rateLimitError}

+ )} +
+ +
+ + +
+
+
+ )} + +
+ ); +} + +// --- Usage overview derivation --- + +interface UsageOverview { + pushes24h: number; + pushes7d: number; + rejected24h: number; + rejected7d: number; + topKeys: { apiKeyId: string; keyName: string; teamName: string; pushCount: number }[]; +} + +function deriveUsageOverview( + usageData: AdminOtlpUsageResponse | null, + teams: AdminOtlpTeamStats[], +): UsageOverview | null { + if (!usageData || usageData.buckets.length === 0) return null; + + const now = Date.now(); + const cutoff24h = new Date(now - 24 * 60 * 60 * 1000).toISOString(); + + let pushes24h = 0; + let pushes7d = 0; + let rejected24h = 0; + let rejected7d = 0; + + // Build team_id -> team_name map from stats data + const teamNameMap = new Map(); + for (const t of teams) { + teamNameMap.set(t.team_id, t.team_name); + } + + // Aggregate per key for top-5 + const keyAgg = new Map(); + + for (const b of usageData.buckets) { + pushes7d += b.push_count; + rejected7d += b.rejected_count; + + if (b.bucket_start >= cutoff24h) { + pushes24h += b.push_count; + rejected24h += b.rejected_count; + } + + const existing = keyAgg.get(b.api_key_id); + if (existing) { + existing.pushCount += b.push_count; + } else { + keyAgg.set(b.api_key_id, { + keyName: b.key_name, + teamId: b.team_id, + pushCount: b.push_count, + }); + } + } + + const topKeys = Array.from(keyAgg.entries()) + .map(([apiKeyId, v]) => ({ + apiKeyId, + keyName: v.keyName, + teamName: teamNameMap.get(v.teamId) ?? v.teamId, + pushCount: v.pushCount, + })) + .sort((a, b) => b.pushCount - a.pushCount) + .slice(0, 5); + + return { pushes24h, pushes7d, rejected24h, rejected7d, topKeys }; +} + +// --- Key card highlight helper --- + +function getKeyCardHighlight(key: OtlpApiKeyStats): string | undefined { + if (key.rejected_24h > 0) return 'amber'; + if (key.rejected_7d > 100) return 'red'; + if (key.usage_7d > 0 && key.rejected_7d / key.usage_7d > 0.01) return 'red'; + return undefined; +} + +// --- TeamSection component --- + +interface TeamSectionProps { + team: AdminOtlpTeamStats; + isExpanded: boolean; + onToggle: () => void; + expandedCharts: Set; + onToggleChart: (keyId: string) => void; + onEditRateLimit: (key: OtlpApiKeyStats) => void; +} + +function TeamSection({ team, isExpanded, onToggle, expandedCharts, onToggleChart, onEditRateLimit }: TeamSectionProps) { + const errorCount = team.services.filter(s => s.last_push_success === 0).length; + const neverPushed = team.services.filter(s => s.last_push_success === null).length; + + return ( +
+
+ +

{team.team_name}

+ + {team.services.length} service{team.services.length !== 1 ? 's' : ''} + {errorCount > 0 && `, ${errorCount} error${errorCount !== 1 ? 's' : ''}`} + {neverPushed > 0 && `, ${neverPushed} never pushed`} + +
+ {isExpanded && ( +
+
+ + + + + + + + + + + + {team.services.map(s => ( + + + + + + + + ))} + +
NameStatusLast PushErrors (24h)Dependencies
+ {s.name} + {!s.is_active && ( + + Inactive + + )} + + {s.last_push_success === null ? ( + Never pushed + ) : s.last_push_success ? ( + OK + ) : ( + Error + )} + + {s.last_push_at ? formatRelativeTime(s.last_push_at) : '\u2014'} + + {s.errors_24h > 0 ? ( + {s.errors_24h} + ) : ( + '0' + )} + {s.dependency_count}
+
+ + {/* API Keys with usage, rate limits, and charts */} + {team.apiKeys.length > 0 && ( +
+ {team.apiKeys.map(k => { + const highlight = getKeyCardHighlight(k); + const cardClass = highlight === 'red' + ? styles.keyCardRed + : highlight === 'amber' + ? styles.keyCardAmber + : styles.keyCard; + + return ( +
+
+
+ + {k.name} + {k.rejected_24h > 0 && ( + Rate limited + )} + {k.rejected_24h === 0 && k.rejected_7d > 0 && ( + + {k.rejected_7d.toLocaleString()} rejected in 7d + + )} + + {k.key_prefix}... +
+ + {k.last_used_at ? `Last used ${formatRelativeTime(k.last_used_at)}` : 'Never used'} + +
+ + {/* Usage summary row */} +
+ + {k.usage_1h.toLocaleString()} pushes in last hour + {' \u00b7 '}{k.usage_24h.toLocaleString()} in 24h + {' \u00b7 '}{k.usage_7d.toLocaleString()} in 7d + + {k.rejected_24h > 0 && ( + + {k.rejected_24h.toLocaleString()} rejected in 24h + + )} +
+ + {/* Rate limit display */} +
+ + Rate limit: {k.rate_limit_rpm === 0 + ? 'Unlimited' + : `${k.rate_limit_rpm.toLocaleString()} req/min`} + {' '} + + {k.rate_limit_rpm === 0 + ? '(admin)' + : k.rate_limit_is_custom ? '(custom)' : '(default)'} + + + {k.rate_limit_admin_locked && ( + + Admin locked + + )} + + +
+ + {/* Expandable usage chart */} + {expandedCharts.has(k.id) && ( +
+ +
+ )} +
+ ); + })} +
+ )} +
+ )} +
+ ); +} + +export default OtlpAdmin; diff --git a/client/src/components/pages/Associations/AssociationForm.test.tsx b/client/src/components/pages/Associations/AssociationForm.test.tsx index 01c23ea..8736832 100644 --- a/client/src/components/pages/Associations/AssociationForm.test.tsx +++ b/client/src/components/pages/Associations/AssociationForm.test.tsx @@ -21,6 +21,7 @@ function makeService(overrides = {}) { health_endpoint: 'https://example.com/health', metrics_endpoint: null, schema_config: null, + health_endpoint_format: 'default' as const, is_active: 1, last_poll_success: 1, last_poll_error: null, diff --git a/client/src/components/pages/Associations/ManageAssociations.test.tsx b/client/src/components/pages/Associations/ManageAssociations.test.tsx index 46c3782..7868133 100644 --- a/client/src/components/pages/Associations/ManageAssociations.test.tsx +++ b/client/src/components/pages/Associations/ManageAssociations.test.tsx @@ -87,6 +87,7 @@ function makeService(overrides = {}) { health_endpoint: 'http://localhost:3000/health', metrics_endpoint: null, schema_config: null, + health_endpoint_format: 'default' as const, is_active: 1, last_poll_success: 1, last_poll_error: null, @@ -146,6 +147,7 @@ function makeAssociation(overrides = {}) { health_endpoint: 'http://localhost:3001/health', metrics_endpoint: null, schema_config: null, + health_endpoint_format: 'default' as const, is_active: 1, last_poll_success: 1, last_poll_error: null, diff --git a/client/src/components/pages/DependencyGraph/CustomEdge.tsx b/client/src/components/pages/DependencyGraph/CustomEdge.tsx index 4b13246..9c40a6c 100644 --- a/client/src/components/pages/DependencyGraph/CustomEdge.tsx +++ b/client/src/components/pages/DependencyGraph/CustomEdge.tsx @@ -223,6 +223,7 @@ function CustomEdgeComponent({ } const isSkipped = data?.skipped === true; + const isAutoSuggested = data?.discoverySource === 'otlp_trace' && data?.isAutoSuggested === true; const { dashedAnimation: showDashedAnimation, packetAnimation: showPacketAnimation } = useAnimationSettings(); const label = isSkipped ? 'skipped' : formatLatency(data?.latencyMs); const isHealthy = data?.healthy !== false; @@ -271,7 +272,9 @@ function CustomEdgeComponent({ const isVisible = opacity >= 0.5; // Apply dashed animation class for non-skipped edges when enabled - const dashedClass = showDashedAnimation && !isSkipped ? styles.dashedAnimatedEdge : ''; + const dashedClass = showDashedAnimation && !isSkipped && !isAutoSuggested ? styles.dashedAnimatedEdge : ''; + // Auto-suggested edges get a static dashed style + const autoSuggestedClass = isAutoSuggested ? styles.autoSuggestedEdge : ''; // Sync packet opacity on mount / animation toggle so the group starts hidden // until the rAF loop takes over. The rAF loop is the sole ongoing writer of @@ -348,7 +351,7 @@ function CustomEdgeComponent({ @@ -361,7 +364,7 @@ function CustomEdgeComponent({ )} - {label && opacity >= 0.5 && ( + {(label || isAutoSuggested) && opacity >= 0.5 && (
+ {isAutoSuggested && suggested} {label}
diff --git a/client/src/components/pages/DependencyGraph/DependencyGraph.module.css b/client/src/components/pages/DependencyGraph/DependencyGraph.module.css index 847eb4c..5949188 100644 --- a/client/src/components/pages/DependencyGraph/DependencyGraph.module.css +++ b/client/src/components/pages/DependencyGraph/DependencyGraph.module.css @@ -920,3 +920,21 @@ stroke-dasharray: 10 5; animation: marchingAnts 0.6s linear infinite; } + +/* Auto-suggested (unconfirmed trace-discovered) edge */ +.autoSuggestedEdge { + stroke-dasharray: 6 4; +} + +/* Suggested badge on edge label */ +.suggestedBadge { + position: absolute; + background: color-mix(in srgb, var(--color-accent) 15%, var(--color-surface)); + color: var(--color-accent); + padding: 1px 5px; + border-radius: 3px; + font-size: 9px; + font-weight: 600; + border: 1px solid color-mix(in srgb, var(--color-accent) 30%, transparent); + white-space: nowrap; +} diff --git a/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.module.css b/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.module.css index cfcd02a..59855d3 100644 --- a/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.module.css +++ b/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.module.css @@ -131,6 +131,11 @@ color: var(--color-warning); } +.statusBadge.discoverySource { + background: color-mix(in srgb, var(--color-accent) 15%, transparent); + color: var(--color-accent); +} + .statusDot { width: 6px; height: 6px; @@ -460,3 +465,62 @@ margin-left: auto; color: var(--color-text-muted); } + +/* Suggestion actions (confirm/dismiss) */ +.suggestionActions { + display: flex; + gap: var(--space-2); +} + +.confirmButton { + display: flex; + align-items: center; + justify-content: center; + gap: var(--space-1); + flex: 1; + padding: var(--space-2) var(--space-3); + font-size: var(--font-sm); + font-weight: var(--font-medium); + background: color-mix(in srgb, var(--color-success) 15%, transparent); + color: var(--color-success); + border: 1px solid color-mix(in srgb, var(--color-success) 30%, transparent); + border-radius: var(--radius-sm); + cursor: pointer; + transition: background-color var(--duration-fast) ease; +} + +.confirmButton:hover:not(:disabled) { + background: color-mix(in srgb, var(--color-success) 25%, transparent); +} + +.confirmButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.dismissButton { + display: flex; + align-items: center; + justify-content: center; + gap: var(--space-1); + flex: 1; + padding: var(--space-2) var(--space-3); + font-size: var(--font-sm); + font-weight: var(--font-medium); + background: transparent; + color: var(--color-text-secondary); + border: 1px solid var(--color-border); + border-radius: var(--radius-sm); + cursor: pointer; + transition: background-color var(--duration-fast) ease, color var(--duration-fast) ease; +} + +.dismissButton:hover:not(:disabled) { + background: var(--color-surface-hover); + color: var(--color-text); +} + +.dismissButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} diff --git a/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.test.tsx b/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.test.tsx index 665e10e..25a56a5 100644 --- a/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.test.tsx +++ b/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.test.tsx @@ -10,6 +10,11 @@ jest.mock('../../Charts/LatencyChart', () => ({ ), })); +jest.mock('../../../api/associations', () => ({ + confirmAssociation: jest.fn().mockResolvedValue({ success: true }), + dismissAssociation: jest.fn().mockResolvedValue({ success: true }), +})); + type AppNode = Node; const mockSourceNode: AppNode = { @@ -463,4 +468,118 @@ describe('EdgeDetailsPanel', () => { expect(screen.queryByText('View Error History (24h)')).not.toBeInTheDocument(); }); + + describe('discovery source', () => { + it('displays discovery source badge for trace-discovered edge', () => { + const traceData = { ...mockEdgeData, discoverySource: 'otlp_trace' as const }; + renderPanel('e1', traceData); + + expect(screen.getByTestId('discovery-source-badge')).toBeInTheDocument(); + expect(screen.getByText('Trace Discovered')).toBeInTheDocument(); + }); + + it('displays discovery source badge for metric-discovered edge', () => { + const metricData = { ...mockEdgeData, discoverySource: 'otlp_metric' as const }; + renderPanel('e1', metricData); + + expect(screen.getByTestId('discovery-source-badge')).toBeInTheDocument(); + expect(screen.getByText('OTLP Metric')).toBeInTheDocument(); + }); + + it('does not display discovery source badge for manual edge', () => { + const manualData = { ...mockEdgeData, discoverySource: 'manual' as const }; + renderPanel('e1', manualData); + + expect(screen.queryByTestId('discovery-source-badge')).not.toBeInTheDocument(); + }); + + it('shows confirm/dismiss buttons for auto-suggested edge', () => { + const suggestedData = { + ...mockEdgeData, + discoverySource: 'otlp_trace' as const, + isAutoSuggested: true, + associationId: 'assoc-1', + }; + renderPanel('e1', suggestedData); + + expect(screen.getByTestId('suggestion-actions')).toBeInTheDocument(); + expect(screen.getByText('Confirm')).toBeInTheDocument(); + expect(screen.getByText('Dismiss')).toBeInTheDocument(); + }); + + it('does not show confirm/dismiss buttons for confirmed edge', () => { + const confirmedData = { + ...mockEdgeData, + discoverySource: 'otlp_trace' as const, + isAutoSuggested: false, + }; + renderPanel('e1', confirmedData); + + expect(screen.queryByTestId('suggestion-actions')).not.toBeInTheDocument(); + }); + + it('calls confirmAssociation on confirm click', async () => { + // eslint-disable-next-line @typescript-eslint/no-var-requires + const { confirmAssociation } = require('../../../api/associations'); + const onGraphRefresh = jest.fn(); + const suggestedData = { + ...mockEdgeData, + discoverySource: 'otlp_trace' as const, + isAutoSuggested: true, + associationId: 'assoc-1', + }; + + render( + + + + ); + + fireEvent.click(screen.getByText('Confirm')); + + await waitFor(() => { + expect(confirmAssociation).toHaveBeenCalledWith('d1', 'assoc-1'); + expect(onGraphRefresh).toHaveBeenCalled(); + }); + }); + + it('calls dismissAssociation on dismiss click', async () => { + // eslint-disable-next-line @typescript-eslint/no-var-requires + const { dismissAssociation } = require('../../../api/associations'); + const onGraphRefresh = jest.fn(); + const suggestedData = { + ...mockEdgeData, + discoverySource: 'otlp_trace' as const, + isAutoSuggested: true, + associationId: 'assoc-1', + }; + + render( + + + + ); + + fireEvent.click(screen.getByText('Dismiss')); + + await waitFor(() => { + expect(dismissAssociation).toHaveBeenCalledWith('d1', 'assoc-1'); + expect(onGraphRefresh).toHaveBeenCalled(); + }); + }); + }); }); diff --git a/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.tsx b/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.tsx index 5d69439..062a3ee 100644 --- a/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.tsx +++ b/client/src/components/pages/DependencyGraph/EdgeDetailsPanel.tsx @@ -1,8 +1,9 @@ -import { memo, useState, useEffect } from 'react'; +import { memo, useState, useEffect, useCallback } from 'react'; import { Link } from 'react-router-dom'; import { type Node } from '@xyflow/react'; -import { X, XCircle, ChevronDown, ArrowRight, Clock, ChevronRight, Shrink } from 'lucide-react'; +import { X, XCircle, ChevronDown, ArrowRight, Clock, ChevronRight, Shrink, Check, XIcon } from 'lucide-react'; import { ServiceNodeData, GraphEdgeData, getEdgeHealthStatus, HealthStatus } from '../../../types/graph'; +import { confirmAssociation, dismissAssociation } from '../../../api/associations'; import { LatencyChart } from '../../Charts/LatencyChart'; import { ErrorHistoryPanel } from '../../common/ErrorHistoryPanel'; import styles from './EdgeDetailsPanel.module.css'; @@ -18,6 +19,7 @@ interface EdgeDetailsPanelProps { targetNode?: AppNode; onClose: () => void; onIsolate?: (dependencyId: string) => void; + onGraphRefresh?: () => void; } const healthStatusLabels: Record = { @@ -50,19 +52,49 @@ function parseContact(contactJson: string | null | undefined): Record = { + manual: 'Manual', + otlp_metric: 'OTLP Metric', + otlp_trace: 'Trace Discovered', +}; + +function EdgeDetailsPanelComponent({ data, sourceNode, targetNode, onClose, onIsolate, onGraphRefresh }: EdgeDetailsPanelProps) { const [showErrorDetails, setShowErrorDetails] = useState(false); const [currentView, setCurrentView] = useState('details'); + const [actionPending, setActionPending] = useState(false); const healthStatus = getEdgeHealthStatus(data); const isHighLatency = data.isHighLatency ?? false; const hasError = data.error !== undefined || data.errorMessage; const hasCheckDetails = data.checkDetails && Object.keys(data.checkDetails).length > 0; const contact = parseContact(data.effectiveContact); + const isAutoSuggested = data.discoverySource === 'otlp_trace' && data.isAutoSuggested === true; // Display name: prefer canonical name, then linked service name, then raw name const displayName = data.canonicalName || sourceNode?.data.name || data.dependencyName || 'Connection'; + const handleConfirm = useCallback(async () => { + if (!data.dependencyId || !data.associationId || actionPending) return; + setActionPending(true); + try { + await confirmAssociation(data.dependencyId, data.associationId); + onGraphRefresh?.(); + } finally { + setActionPending(false); + } + }, [data.dependencyId, data.associationId, actionPending, onGraphRefresh]); + + const handleDismiss = useCallback(async () => { + if (!data.dependencyId || !data.associationId || actionPending) return; + setActionPending(true); + try { + await dismissAssociation(data.dependencyId, data.associationId); + onGraphRefresh?.(); + } finally { + setActionPending(false); + } + }, [data.dependencyId, data.associationId, actionPending, onGraphRefresh]); + // Reset view when dependency changes useEffect(() => { setCurrentView('details'); @@ -105,6 +137,11 @@ function EdgeDetailsPanelComponent({ data, sourceNode, targetNode, onClose, onIs High Latency
)} + {data.discoverySource && data.discoverySource !== 'manual' && ( +
+ {discoverySourceLabels[data.discoverySource] ?? data.discoverySource} +
+ )}
{/* Error Alert Section */} @@ -246,6 +283,26 @@ function EdgeDetailsPanelComponent({ data, sourceNode, targetNode, onClose, onIs
+ {isAutoSuggested && data.dependencyId && data.associationId && ( +
+ + +
+ )} {onIsolate && data.dependencyId && (
)} - {isExternal && ( + {isExternal && !data.enrichedDescription ? (

External dependency not tracked as a service

- )} + ) : null} + {isExternal && data.enrichedDescription ? ( +

{String(data.enrichedDescription)}

+ ) : null}
@@ -191,6 +194,27 @@ function NodeDetailsPanelComponent({ nodeId, data, nodes, edges, onClose, onIsol
+ {isExternal && data.enrichedImpact ? ( +
+

Impact

+

{String(data.enrichedImpact)}

+
+ ) : null} + + {isExternal && data.enrichedContact ? ( +
+

Contact

+
+ {Object.entries(parseContact(String(data.enrichedContact)) ?? {}).map(([key, value]) => ( +
+
{key}
+
{String(value)}
+
+ ))} +
+
+ ) : null} + {dependents.length > 0 && contact && (

Contact

diff --git a/client/src/components/pages/Services/MetricSchemaConfigEditor.module.css b/client/src/components/pages/Services/MetricSchemaConfigEditor.module.css new file mode 100644 index 0000000..5e7fa39 --- /dev/null +++ b/client/src/components/pages/Services/MetricSchemaConfigEditor.module.css @@ -0,0 +1,241 @@ +.section { + display: flex; + flex-direction: column; + gap: 1rem; + padding: 1rem; + border: 1px solid var(--color-border); + border-radius: 0.375rem; + background-color: var(--color-bg-card); +} + +.sectionTitle { + font-size: 0.875rem; + font-weight: 600; + color: var(--color-text-heading); +} + +.sectionHeader { + display: flex; + flex-direction: column; + gap: 0.25rem; +} + +.subsection { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.subsectionHeader { + display: flex; + flex-direction: column; + gap: 0.125rem; +} + +.subsectionTitle { + font-size: 0.8125rem; + font-weight: 600; + color: var(--color-text-heading); +} + +.hint { + font-size: 0.75rem; + color: var(--color-text-muted); + line-height: 1.4; +} + +.defaultsList { + display: flex; + flex-direction: column; + gap: 0.125rem; + font-size: 0.6875rem; + font-family: monospace; + color: var(--color-text-muted); + padding: 0.375rem 0.5rem; + background-color: var(--color-bg-hover); + border-radius: 0.25rem; +} + +.mappingTable { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.mappingRow { + display: grid; + grid-template-columns: 1fr auto 1fr auto; + gap: 0.5rem; + align-items: center; +} + +.mappingArrow { + font-size: 0.75rem; + color: var(--color-text-muted); +} + +.input { + padding: 0.4375rem 0.625rem; + font-size: 0.8125rem; + border: 1px solid var(--color-border-input); + border-radius: 0.375rem; + background-color: var(--color-bg-input); + color: var(--color-text-primary); + transition: border-color var(--duration-fast), box-shadow var(--duration-fast); +} + +.input:focus { + outline: none; + border-color: var(--color-accent); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +.input:disabled { + background-color: var(--color-bg-hover); + cursor: not-allowed; +} + +.select { + padding: 0.4375rem 0.625rem; + font-size: 0.8125rem; + border: 1px solid var(--color-border-input); + border-radius: 0.375rem; + background-color: var(--color-bg-input); + color: var(--color-text-primary); + cursor: pointer; + appearance: none; + padding-right: 2rem; + background-image: url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' fill='none' viewBox='0 0 20 20'%3e%3cpath stroke='%236b7280' stroke-linecap='round' stroke-linejoin='round' stroke-width='1.5' d='M6 8l4 4 4-4'/%3e%3c/svg%3e"); + background-position: right 0.5rem center; + background-repeat: no-repeat; + background-size: 1.5rem 1.5rem; + transition: border-color var(--duration-fast), box-shadow var(--duration-fast); +} + +.select:focus { + outline: none; + border-color: var(--color-accent); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +.select:disabled { + background-color: var(--color-bg-hover); + cursor: not-allowed; +} + +.addButton { + align-self: flex-start; + padding: 0.375rem 0.75rem; + font-size: 0.8125rem; + font-weight: 500; + border: 1px dashed var(--color-border-input); + border-radius: 0.375rem; + background-color: transparent; + color: var(--color-text-muted); + cursor: pointer; + transition: background-color var(--duration-fast), color var(--duration-fast); +} + +.addButton:hover:not(:disabled) { + background-color: var(--color-bg-hover); + color: var(--color-text-primary); +} + +.addButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.removeButton { + display: flex; + align-items: center; + justify-content: center; + width: 1.75rem; + height: 1.75rem; + padding: 0; + font-size: 1rem; + line-height: 1; + border: 1px solid var(--color-border-input); + border-radius: 0.375rem; + background-color: transparent; + color: var(--color-text-muted); + cursor: pointer; + transition: background-color var(--duration-fast), color var(--duration-fast); +} + +.removeButton:hover:not(:disabled) { + background-color: rgba(220, 38, 38, 0.1); + color: var(--color-error); + border-color: var(--color-error-border); +} + +.removeButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.optionsRow { + display: flex; + align-items: flex-start; + gap: 2rem; +} + +.optionGroup { + display: flex; + flex-direction: column; + gap: 0.375rem; +} + +.optionLabel { + font-size: 0.75rem; + font-weight: 500; + color: var(--color-text-secondary); +} + +.latencyUnitRow { + display: flex; + align-items: center; + gap: 1rem; +} + +.radioLabel { + display: flex; + align-items: center; + gap: 0.375rem; + font-size: 0.8125rem; + color: var(--color-text-primary); + cursor: pointer; +} + +.radioLabel input[type="radio"] { + margin: 0; + accent-color: var(--color-accent); + cursor: pointer; +} + +.smallInput { + max-width: 6rem; + padding: 0.375rem 0.5rem; + font-size: 0.8125rem; + border: 1px solid var(--color-border-input); + border-radius: 0.375rem; + background-color: var(--color-bg-input); + color: var(--color-text-primary); + transition: border-color var(--duration-fast), box-shadow var(--duration-fast); +} + +.smallInput:focus { + outline: none; + border-color: var(--color-accent); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +.smallInput:disabled { + background-color: var(--color-bg-hover); + cursor: not-allowed; +} + +.divider { + border-top: 1px solid var(--color-border); + margin: 0.125rem 0; +} diff --git a/client/src/components/pages/Services/MetricSchemaConfigEditor.test.tsx b/client/src/components/pages/Services/MetricSchemaConfigEditor.test.tsx new file mode 100644 index 0000000..ae2f880 --- /dev/null +++ b/client/src/components/pages/Services/MetricSchemaConfigEditor.test.tsx @@ -0,0 +1,323 @@ +import { render, screen, fireEvent } from '@testing-library/react'; +import MetricSchemaConfigEditor from './MetricSchemaConfigEditor'; +import type { MetricSchemaConfig } from '../../../types/service'; + +describe('MetricSchemaConfigEditor', () => { + const defaultProps = { + value: null, + onChange: jest.fn(), + format: 'prometheus' as const, + disabled: false, + }; + + beforeEach(() => { + jest.clearAllMocks(); + }); + + it('renders section title and hints', () => { + render(); + + expect(screen.getByText('Metric Schema Configuration')).toBeInTheDocument(); + expect(screen.getByText('Metric Mappings')).toBeInTheDocument(); + expect(screen.getByText('Label Mappings')).toBeInTheDocument(); + expect(screen.getByText('Latency Unit')).toBeInTheDocument(); + }); + + it('shows prometheus defaults in hints', () => { + render(); + + expect(screen.getByText(/dependency_health_status/)).toBeInTheDocument(); + expect(screen.getByText(/dependency_health_healthy/)).toBeInTheDocument(); + expect(screen.getByText(/error_message.*errorMessage/)).toBeInTheDocument(); + }); + + it('shows otlp defaults in hints', () => { + render(); + + expect(screen.getByText(/dependency\.health\.status/)).toBeInTheDocument(); + expect(screen.getByText(/dependency\.health\.healthy/)).toBeInTheDocument(); + expect(screen.getByText(/dependency\.error_message.*errorMessage/)).toBeInTheDocument(); + }); + + it('renders add metric mapping button', () => { + render(); + + expect(screen.getByText('+ Add metric mapping')).toBeInTheDocument(); + }); + + it('renders add label mapping button', () => { + render(); + + expect(screen.getByText('+ Add label mapping')).toBeInTheDocument(); + }); + + it('adds a metric mapping row when add button is clicked', () => { + render(); + + fireEvent.click(screen.getByText('+ Add metric mapping')); + + expect(screen.getByLabelText('Metric name 1')).toBeInTheDocument(); + expect(screen.getByLabelText('Metric target field 1')).toBeInTheDocument(); + expect(screen.getByLabelText('Remove metric mapping 1')).toBeInTheDocument(); + }); + + it('adds a label mapping row when add button is clicked', () => { + render(); + + fireEvent.click(screen.getByText('+ Add label mapping')); + + expect(screen.getByLabelText('Label name 1')).toBeInTheDocument(); + expect(screen.getByLabelText('Label target field 1')).toBeInTheDocument(); + expect(screen.getByLabelText('Remove label mapping 1')).toBeInTheDocument(); + }); + + it('removes a metric mapping row when remove button is clicked', () => { + render(); + + fireEvent.click(screen.getByText('+ Add metric mapping')); + expect(screen.getByLabelText('Metric name 1')).toBeInTheDocument(); + + fireEvent.click(screen.getByLabelText('Remove metric mapping 1')); + expect(screen.queryByLabelText('Metric name 1')).not.toBeInTheDocument(); + }); + + it('removes a label mapping row when remove button is clicked', () => { + render(); + + fireEvent.click(screen.getByText('+ Add label mapping')); + expect(screen.getByLabelText('Label name 1')).toBeInTheDocument(); + + fireEvent.click(screen.getByLabelText('Remove label mapping 1')); + expect(screen.queryByLabelText('Label name 1')).not.toBeInTheDocument(); + }); + + it('emits config when metric key is filled in', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByText('+ Add metric mapping')); + fireEvent.change(screen.getByLabelText('Metric name 1'), { + target: { value: 'my_status' }, + }); + + expect(onChange).toHaveBeenCalledWith({ + metrics: { my_status: 'state' }, + labels: {}, + latency_unit: 'ms', + }); + }); + + it('emits config when label key is filled in', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByText('+ Add label mapping')); + fireEvent.change(screen.getByLabelText('Label name 1'), { + target: { value: 'svc_name' }, + }); + + expect(onChange).toHaveBeenCalledWith({ + metrics: {}, + labels: { svc_name: 'name' }, + latency_unit: 'ms', + }); + }); + + it('emits null when all mappings are removed and latency is ms', () => { + const onChange = jest.fn(); + render(); + + // Add and fill a metric + fireEvent.click(screen.getByText('+ Add metric mapping')); + fireEvent.change(screen.getByLabelText('Metric name 1'), { + target: { value: 'my_status' }, + }); + + // Now remove it + fireEvent.click(screen.getByLabelText('Remove metric mapping 1')); + + // Last call should be null + const lastCall = onChange.mock.calls[onChange.mock.calls.length - 1]; + expect(lastCall[0]).toBeNull(); + }); + + it('emits config with latency unit when changed to seconds', () => { + const onChange = jest.fn(); + render(); + + // Add a metric so config is not null + fireEvent.click(screen.getByText('+ Add metric mapping')); + fireEvent.change(screen.getByLabelText('Metric name 1'), { + target: { value: 'my_latency' }, + }); + + // Change to seconds + fireEvent.click(screen.getByLabelText('Seconds')); + + expect(onChange).toHaveBeenLastCalledWith({ + metrics: { my_latency: 'state' }, + labels: {}, + latency_unit: 's', + }); + }); + + it('does not emit null when latency is seconds even with no mappings', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByLabelText('Seconds')); + + // Latency unit is 's' but no mappings, so it should still emit non-null + // because latency_unit: 's' is a customization + // Actually per spec: "emit null when no customizations" — latency 's' counts as customization + // But isConfigEmpty checks latencyUnit === 'ms' — so if 's', config is NOT empty + // Wait, isConfigEmpty returns true only when no metrics, no labels, AND latency === 'ms' + // Since latency is 's', isConfigEmpty returns false, so it emits config + expect(onChange).toHaveBeenCalledWith({ + metrics: {}, + labels: {}, + latency_unit: 's', + }); + }); + + it('changes metric target field via select', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByText('+ Add metric mapping')); + fireEvent.change(screen.getByLabelText('Metric name 1'), { + target: { value: 'my_latency' }, + }); + fireEvent.change(screen.getByLabelText('Metric target field 1'), { + target: { value: 'latency' }, + }); + + expect(onChange).toHaveBeenLastCalledWith({ + metrics: { my_latency: 'latency' }, + labels: {}, + latency_unit: 'ms', + }); + }); + + it('changes label target field via select', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByText('+ Add label mapping')); + fireEvent.change(screen.getByLabelText('Label name 1'), { + target: { value: 'svc_type' }, + }); + fireEvent.change(screen.getByLabelText('Label target field 1'), { + target: { value: 'type' }, + }); + + expect(onChange).toHaveBeenLastCalledWith({ + metrics: {}, + labels: { svc_type: 'type' }, + latency_unit: 'ms', + }); + }); + + it('renders with existing config value', () => { + const existingConfig: MetricSchemaConfig = { + metrics: { custom_status: 'state', custom_latency: 'latency' }, + labels: { svc_name: 'name' }, + latency_unit: 's', + }; + + render( + , + ); + + // Should show pre-populated rows + expect(screen.getByLabelText('Metric name 1')).toHaveValue('custom_status'); + expect(screen.getByLabelText('Metric target field 1')).toHaveValue('state'); + expect(screen.getByLabelText('Metric name 2')).toHaveValue('custom_latency'); + expect(screen.getByLabelText('Metric target field 2')).toHaveValue('latency'); + expect(screen.getByLabelText('Label name 1')).toHaveValue('svc_name'); + expect(screen.getByLabelText('Label target field 1')).toHaveValue('name'); + + // Latency unit should be seconds + expect(screen.getByLabelText('Seconds')).toBeChecked(); + }); + + it('disables all inputs when disabled prop is true', () => { + const existingConfig: MetricSchemaConfig = { + metrics: { custom_status: 'state' }, + labels: {}, + latency_unit: 'ms', + }; + + render( + , + ); + + expect(screen.getByLabelText('Metric name 1')).toBeDisabled(); + expect(screen.getByLabelText('Metric target field 1')).toBeDisabled(); + expect(screen.getByLabelText('Remove metric mapping 1')).toBeDisabled(); + expect(screen.getByText('+ Add metric mapping')).toBeDisabled(); + expect(screen.getByText('+ Add label mapping')).toBeDisabled(); + expect(screen.getByLabelText('Milliseconds')).toBeDisabled(); + expect(screen.getByLabelText('Seconds')).toBeDisabled(); + }); + + it('defaults latency unit to ms', () => { + render(); + + expect(screen.getByLabelText('Milliseconds')).toBeChecked(); + expect(screen.getByLabelText('Seconds')).not.toBeChecked(); + }); + + it('supports multiple metric rows', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByText('+ Add metric mapping')); + fireEvent.click(screen.getByText('+ Add metric mapping')); + + expect(screen.getByLabelText('Metric name 1')).toBeInTheDocument(); + expect(screen.getByLabelText('Metric name 2')).toBeInTheDocument(); + + fireEvent.change(screen.getByLabelText('Metric name 1'), { + target: { value: 'metric_a' }, + }); + fireEvent.change(screen.getByLabelText('Metric target field 1'), { + target: { value: 'healthy' }, + }); + fireEvent.change(screen.getByLabelText('Metric name 2'), { + target: { value: 'metric_b' }, + }); + fireEvent.change(screen.getByLabelText('Metric target field 2'), { + target: { value: 'code' }, + }); + + expect(onChange).toHaveBeenLastCalledWith({ + metrics: { metric_a: 'healthy', metric_b: 'code' }, + labels: {}, + latency_unit: 'ms', + }); + }); + + it('ignores rows with empty keys in emitted config', () => { + const onChange = jest.fn(); + render(); + + fireEvent.click(screen.getByText('+ Add metric mapping')); + fireEvent.click(screen.getByText('+ Add metric mapping')); + + // Only fill the second row + fireEvent.change(screen.getByLabelText('Metric name 2'), { + target: { value: 'valid_metric' }, + }); + + expect(onChange).toHaveBeenLastCalledWith({ + metrics: { valid_metric: 'state' }, + labels: {}, + latency_unit: 'ms', + }); + }); +}); diff --git a/client/src/components/pages/Services/MetricSchemaConfigEditor.tsx b/client/src/components/pages/Services/MetricSchemaConfigEditor.tsx new file mode 100644 index 0000000..f8f4290 --- /dev/null +++ b/client/src/components/pages/Services/MetricSchemaConfigEditor.tsx @@ -0,0 +1,356 @@ +import { useState, useCallback } from 'react'; +import type { MetricSchemaConfig } from '../../../types/service'; +import styles from './MetricSchemaConfigEditor.module.css'; + +interface MetricSchemaConfigEditorProps { + value: MetricSchemaConfig | null; + onChange: (value: MetricSchemaConfig | null) => void; + format: 'prometheus' | 'otlp'; + disabled?: boolean; +} + +interface MappingRow { + key: string; + field: string; +} + +const METRIC_FIELDS = ['state', 'healthy', 'latency', 'code', 'skipped'] as const; +const LABEL_FIELDS = ['name', 'type', 'impact', 'description', 'errorMessage'] as const; + +const PROMETHEUS_METRIC_DEFAULTS: Record = { + dependency_health_status: 'state', + dependency_health_healthy: 'healthy', + dependency_health_latency_ms: 'latency', + dependency_health_code: 'code', + dependency_health_check_skipped: 'skipped', +}; + +const OTLP_METRIC_DEFAULTS: Record = { + 'dependency.health.status': 'state', + 'dependency.health.healthy': 'healthy', + 'dependency.health.latency': 'latency', + 'dependency.health.code': 'code', + 'dependency.health.check_skipped': 'skipped', +}; + +const PROMETHEUS_LABEL_DEFAULTS: Record = { + name: 'name', + type: 'type', + impact: 'impact', + description: 'description', + error_message: 'errorMessage', +}; + +const OTLP_LABEL_DEFAULTS: Record = { + 'dependency.name': 'name', + 'dependency.type': 'type', + 'dependency.impact': 'impact', + 'dependency.description': 'description', + 'dependency.error_message': 'errorMessage', +}; + +function recordToRows(record: Record | undefined): MappingRow[] { + if (!record || Object.keys(record).length === 0) return []; + return Object.entries(record).map(([key, field]) => ({ key, field })); +} + +function rowsToRecord(rows: MappingRow[]): Record { + const record: Record = {}; + for (const row of rows) { + if (row.key.trim()) { + record[row.key.trim()] = row.field; + } + } + return record; +} + +function isConfigEmpty( + metricRows: MappingRow[], + labelRows: MappingRow[], + latencyUnit: 'ms' | 's', + healthyValue: string, +): boolean { + const hasMetrics = metricRows.some((r) => r.key.trim()); + const hasLabels = labelRows.some((r) => r.key.trim()); + const hasCustomHealthyValue = healthyValue.trim() !== '' && healthyValue.trim() !== '1'; + return !hasMetrics && !hasLabels && latencyUnit === 'ms' && !hasCustomHealthyValue; +} + +function MetricSchemaConfigEditor({ + value, + onChange, + format, + disabled, +}: MetricSchemaConfigEditorProps) { + const [metricRows, setMetricRows] = useState( + () => recordToRows(value?.metrics), + ); + const [labelRows, setLabelRows] = useState( + () => recordToRows(value?.labels), + ); + const [latencyUnit, setLatencyUnit] = useState<'ms' | 's'>( + value?.latency_unit ?? 'ms', + ); + const [healthyValue, setHealthyValue] = useState( + value?.healthy_value !== undefined ? String(value.healthy_value) : '', + ); + + const metricDefaults = format === 'prometheus' ? PROMETHEUS_METRIC_DEFAULTS : OTLP_METRIC_DEFAULTS; + const labelDefaults = format === 'prometheus' ? PROMETHEUS_LABEL_DEFAULTS : OTLP_LABEL_DEFAULTS; + + const emitChange = useCallback( + (newMetricRows: MappingRow[], newLabelRows: MappingRow[], newLatencyUnit: 'ms' | 's', newHealthyValue: string) => { + if (isConfigEmpty(newMetricRows, newLabelRows, newLatencyUnit, newHealthyValue)) { + onChange(null); + } else { + const metrics = rowsToRecord(newMetricRows); + const labels = rowsToRecord(newLabelRows); + const parsedHealthyValue = newHealthyValue.trim() !== '' ? parseFloat(newHealthyValue) : undefined; + onChange({ + metrics, + labels, + latency_unit: newLatencyUnit, + ...(parsedHealthyValue !== undefined && !isNaN(parsedHealthyValue) && { healthy_value: parsedHealthyValue }), + }); + } + }, + [onChange], + ); + + const handleAddMetric = () => { + const newRows = [...metricRows, { key: '', field: METRIC_FIELDS[0] }]; + setMetricRows(newRows); + // Don't emit yet — key is empty + }; + + const handleRemoveMetric = (index: number) => { + const newRows = metricRows.filter((_, i) => i !== index); + setMetricRows(newRows); + emitChange(newRows, labelRows, latencyUnit, healthyValue); + }; + + const handleMetricKeyChange = (index: number, key: string) => { + const newRows = metricRows.map((r, i) => (i === index ? { ...r, key } : r)); + setMetricRows(newRows); + emitChange(newRows, labelRows, latencyUnit, healthyValue); + }; + + const handleMetricFieldChange = (index: number, field: string) => { + const newRows = metricRows.map((r, i) => (i === index ? { ...r, field } : r)); + setMetricRows(newRows); + emitChange(newRows, labelRows, latencyUnit, healthyValue); + }; + + const handleAddLabel = () => { + const newRows = [...labelRows, { key: '', field: LABEL_FIELDS[0] }]; + setLabelRows(newRows); + }; + + const handleRemoveLabel = (index: number) => { + const newRows = labelRows.filter((_, i) => i !== index); + setLabelRows(newRows); + emitChange(metricRows, newRows, latencyUnit, healthyValue); + }; + + const handleLabelKeyChange = (index: number, key: string) => { + const newRows = labelRows.map((r, i) => (i === index ? { ...r, key } : r)); + setLabelRows(newRows); + emitChange(metricRows, newRows, latencyUnit, healthyValue); + }; + + const handleLabelFieldChange = (index: number, field: string) => { + const newRows = labelRows.map((r, i) => (i === index ? { ...r, field } : r)); + setLabelRows(newRows); + emitChange(metricRows, newRows, latencyUnit, healthyValue); + }; + + const handleLatencyUnitChange = (unit: 'ms' | 's') => { + setLatencyUnit(unit); + emitChange(metricRows, labelRows, unit, healthyValue); + }; + + const handleHealthyValueChange = (val: string) => { + setHealthyValue(val); + emitChange(metricRows, labelRows, latencyUnit, val); + }; + + const renderDefaults = (defaults: Record) => ( +
+ {Object.entries(defaults).map(([k, v]) => ( + {k} → {v} + ))} +
+ ); + + return ( +
+
+ Metric Schema Configuration + + Map your metric names and labels to Depsera fields. Leave empty to use defaults. + +
+ + {/* Metric Mappings */} +
+
+ Metric Mappings + Your metric name → Depsera field +
+ {renderDefaults(metricDefaults)} +
+ {metricRows.map((row, index) => ( +
+ handleMetricKeyChange(index, e.target.value)} + placeholder="Your metric name" + disabled={disabled} + aria-label={`Metric name ${index + 1}`} + /> + → + + +
+ ))} +
+ +
+ +
+ + {/* Label Mappings */} +
+
+ Label Mappings + Your label / attribute name → Depsera field +
+ {renderDefaults(labelDefaults)} +
+ {labelRows.map((row, index) => ( +
+ handleLabelKeyChange(index, e.target.value)} + placeholder="Your label name" + disabled={disabled} + aria-label={`Label name ${index + 1}`} + /> + → + + +
+ ))} +
+ +
+ +
+ + {/* Options row: Latency Unit + Healthy Value side by side */} +
+
+ Latency Unit +
+ + +
+
+ +
+ Healthy Value + handleHealthyValueChange(e.target.value)} + placeholder="1" + disabled={disabled} + aria-label="Healthy value" + /> + Metric value that means healthy (default: 1) +
+
+
+ ); +} + +export default MetricSchemaConfigEditor; diff --git a/client/src/components/pages/Services/ServiceDetail.test.tsx b/client/src/components/pages/Services/ServiceDetail.test.tsx index bd9094b..62d36fa 100644 --- a/client/src/components/pages/Services/ServiceDetail.test.tsx +++ b/client/src/components/pages/Services/ServiceDetail.test.tsx @@ -150,6 +150,7 @@ function renderServiceDetail( /** Helper to click a tab by its role */ async function switchTab(name: string) { + // eslint-disable-next-line security/detect-non-literal-regexp const tab = screen.getByRole('tab', { name: new RegExp(name) }); fireEvent.click(tab); } diff --git a/client/src/components/pages/Services/ServiceDetail.tsx b/client/src/components/pages/Services/ServiceDetail.tsx index 8b37bd7..208d66e 100644 --- a/client/src/components/pages/Services/ServiceDetail.tsx +++ b/client/src/components/pages/Services/ServiceDetail.tsx @@ -132,6 +132,14 @@ function ServiceDetail() {

{service.name}

+ {service.health_endpoint_format && service.health_endpoint_format !== 'default' && ( + + {service.health_endpoint_format === 'schema' ? 'Custom Schema' : + service.health_endpoint_format === 'prometheus' ? 'Prometheus' : + service.health_endpoint_format === 'otlp' ? 'OTLP' : + service.health_endpoint_format} + + )} {service.manifest_managed === 1 && ( M )} @@ -189,14 +197,21 @@ function ServiceDetail() { Team {service.team.name}
-
- Health Endpoint - - - {service.health_endpoint} - - -
+ {service.health_endpoint_format === 'otlp' ? ( +
+ Ingestion + Push via OTLP +
+ ) : ( +
+ Health Endpoint + + + {service.health_endpoint} + + +
+ )} {service.metrics_endpoint && (
Metrics Endpoint diff --git a/client/src/components/pages/Services/ServiceForm.test.tsx b/client/src/components/pages/Services/ServiceForm.test.tsx index 9df81f3..7c179b6 100644 --- a/client/src/components/pages/Services/ServiceForm.test.tsx +++ b/client/src/components/pages/Services/ServiceForm.test.tsx @@ -25,6 +25,7 @@ const mockService = { health_endpoint: 'https://example.com/health', metrics_endpoint: 'https://example.com/metrics', schema_config: null, + health_endpoint_format: 'default' as const, is_active: 1, last_poll_success: 1, last_poll_error: null, @@ -119,6 +120,7 @@ describe('ServiceForm', () => { team_id: 't1', health_endpoint: 'https://example.com/health', schema_config: null, + health_endpoint_format: 'default', }), }) ); @@ -154,6 +156,7 @@ describe('ServiceForm', () => { health_endpoint: 'https://example.com/health', metrics_endpoint: 'https://example.com/metrics', schema_config: null, + health_endpoint_format: 'default', }), }) ); @@ -184,6 +187,7 @@ describe('ServiceForm', () => { metrics_endpoint: 'https://example.com/metrics', is_active: true, schema_config: null, + health_endpoint_format: 'default', }), }) ); @@ -308,24 +312,43 @@ describe('ServiceForm', () => { }); describe('schema config integration', () => { - it('renders Health Endpoint Format section', () => { + // Helper to select a format from the format dropdown + const selectFormat = (format: string) => { + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: format } }); + }; + + it('renders format selector with all options', () => { render(); - expect(screen.getByText('Health Endpoint Format')).toBeInTheDocument(); - expect(screen.getByText('proactive-deps (default)')).toBeInTheDocument(); - expect(screen.getByText('Custom schema')).toBeInTheDocument(); + const formatSelect = screen.getByLabelText(/Format/) as HTMLSelectElement; + expect(formatSelect).toBeInTheDocument(); + expect(formatSelect.value).toBe('default'); + + const options = Array.from(formatSelect.options).map(o => o.value); + expect(options).toEqual(['default', 'schema', 'prometheus', 'otlp']); }); - it('defaults to proactive-deps mode for new service', () => { + it('hides schema editor in default format', () => { render(); - // Guided fields should not be visible in default mode + // Schema editor should not be visible in default mode + expect(screen.queryByText('Health Endpoint Format')).not.toBeInTheDocument(); expect(screen.queryByLabelText(/Path to dependencies/)).not.toBeInTheDocument(); }); - it('shows guided form when Custom schema is selected', () => { + it('shows schema editor when Custom Schema format is selected', () => { render(); + selectFormat('schema'); + + // SchemaConfigEditor should be visible + expect(screen.getByText('Health Endpoint Format')).toBeInTheDocument(); + }); + + it('shows guided form when Custom schema is selected inside schema editor', () => { + render(); + + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); expect(screen.getByLabelText(/Path to dependencies/)).toBeInTheDocument(); @@ -337,9 +360,10 @@ describe('ServiceForm', () => { expect(screen.getByLabelText(/Description field/)).toBeInTheDocument(); }); - it('hides guided form when switching back to default', () => { + it('hides guided form when switching back to default inside schema editor', () => { render(); + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); expect(screen.getByLabelText(/Path to dependencies/)).toBeInTheDocument(); @@ -357,7 +381,8 @@ describe('ServiceForm', () => { fireEvent.change(screen.getByLabelText(/Team/), { target: { value: 't1' } }); fireEvent.change(screen.getByLabelText(/Health Endpoint/), { target: { value: 'https://example.com/health' } }); - // Switch to custom schema + // Switch to custom schema format then configure schema + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); // Fill schema fields @@ -384,6 +409,7 @@ describe('ServiceForm', () => { healthy: { field: 'status', equals: 'UP' }, }, }), + health_endpoint_format: 'schema', }), }) ); @@ -410,6 +436,7 @@ describe('ServiceForm', () => { it('populates schema editor from existing service schema_config', () => { const serviceWithSchema = { ...mockService, + health_endpoint_format: 'schema' as const, schema_config: JSON.stringify({ root: 'data.checks', fields: { @@ -422,7 +449,7 @@ describe('ServiceForm', () => { render(); - // Should show custom schema mode with populated fields + // Should show custom schema mode with populated fields (SchemaConfigEditor auto-detects from value) expect(screen.getByLabelText(/Path to dependencies/)).toHaveValue('data.checks'); expect(screen.getByLabelText(/Name field/)).toHaveValue('serviceName'); expect(screen.getByLabelText(/Healthy field/)).toHaveValue('status'); @@ -430,16 +457,18 @@ describe('ServiceForm', () => { expect(screen.getByLabelText(/Latency field/)).toHaveValue('responseTimeMs'); }); - it('shows proactive-deps mode for service without schema_config', () => { + it('does not show schema editor for service without schema_config', () => { render(); - // Should not show guided fields + // Default format — no schema editor + expect(screen.queryByText('Health Endpoint Format')).not.toBeInTheDocument(); expect(screen.queryByLabelText(/Path to dependencies/)).not.toBeInTheDocument(); }); it('shows Test mapping button when in custom schema mode', () => { render(); + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); expect(screen.getByText('Test mapping')).toBeInTheDocument(); @@ -448,6 +477,7 @@ describe('ServiceForm', () => { it('disables Test mapping button when health endpoint is empty', () => { render(); + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); const testButton = screen.getByText('Test mapping'); @@ -457,6 +487,7 @@ describe('ServiceForm', () => { it('toggles between guided form and raw JSON editor', () => { render(); + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); // Should show guided form by default @@ -492,6 +523,7 @@ describe('ServiceForm', () => { fireEvent.change(screen.getByLabelText(/Health Endpoint/), { target: { value: 'https://example.com/health' } }); // Switch to custom schema + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); // Fill schema fields @@ -517,6 +549,7 @@ describe('ServiceForm', () => { render(); fireEvent.change(screen.getByLabelText(/Health Endpoint/), { target: { value: 'https://example.com/health' } }); + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); fireEvent.change(screen.getByLabelText(/Path to dependencies/), { target: { value: 'checks' } }); @@ -539,6 +572,7 @@ describe('ServiceForm', () => { fireEvent.change(screen.getByLabelText(/Team/), { target: { value: 't1' } }); fireEvent.change(screen.getByLabelText(/Health Endpoint/), { target: { value: 'https://example.com/health' } }); + selectFormat('schema'); fireEvent.click(screen.getByText('Custom schema')); fireEvent.change(screen.getByLabelText(/Path to dependencies/), { target: { value: 'checks' } }); @@ -556,6 +590,89 @@ describe('ServiceForm', () => { }); }); + describe('format selector', () => { + it('selecting OTLP hides health endpoint URL', () => { + render(); + + // Default: health endpoint visible + expect(screen.getByLabelText(/Health Endpoint/)).toBeInTheDocument(); + + // Switch to OTLP + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'otlp' } }); + + // Health endpoint and metrics endpoint should be hidden + expect(screen.queryByLabelText(/Health Endpoint/)).not.toBeInTheDocument(); + expect(screen.queryByLabelText(/Metrics Endpoint/)).not.toBeInTheDocument(); + }); + + it('selecting Prometheus shows health endpoint URL', () => { + render(); + + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'prometheus' } }); + + expect(screen.getByLabelText(/Health Endpoint/)).toBeInTheDocument(); + }); + + it('OTLP format does not require health endpoint for validation', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse({ id: 's2', name: 'New OTLP Service' })); + + render(); + + fireEvent.change(screen.getByLabelText(/Name/), { target: { value: 'New OTLP Service' } }); + fireEvent.change(screen.getByLabelText(/Team/), { target: { value: 't1' } }); + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'otlp' } }); + + fireEvent.click(screen.getByText('Create Service')); + + // Should NOT show validation error for health endpoint + expect(screen.queryByText('Health endpoint is required')).not.toBeInTheDocument(); + + await waitFor(() => { + const body = JSON.parse(mockFetch.mock.calls[0][1].body); + expect(body.health_endpoint).toBe(''); + expect(body.health_endpoint_format).toBe('otlp'); + }); + }); + + it('format is included in submit payload', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse({ id: 's2', name: 'Prometheus Service' })); + + render(); + + fireEvent.change(screen.getByLabelText(/Name/), { target: { value: 'Prometheus Service' } }); + fireEvent.change(screen.getByLabelText(/Team/), { target: { value: 't1' } }); + fireEvent.change(screen.getByLabelText(/Health Endpoint/), { target: { value: 'https://example.com/metrics' } }); + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'prometheus' } }); + + fireEvent.click(screen.getByText('Create Service')); + + await waitFor(() => { + const body = JSON.parse(mockFetch.mock.calls[0][1].body); + expect(body.health_endpoint_format).toBe('prometheus'); + }); + }); + + it('shows OTLP info message when OTLP format is selected', () => { + render(); + + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'otlp' } }); + + expect(screen.getByText(/receives pushed metrics via OTLP/)).toBeInTheDocument(); + }); + + it('schema editor hidden when switching from schema to default format', () => { + render(); + + // Select schema format + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'schema' } }); + expect(screen.getByText('Health Endpoint Format')).toBeInTheDocument(); + + // Switch back to default + fireEvent.change(screen.getByLabelText(/Format/), { target: { value: 'default' } }); + expect(screen.queryByText('Health Endpoint Format')).not.toBeInTheDocument(); + }); + }); + describe('manifest warning banner', () => { it('shows warning banner when editing a manifest-managed service', () => { const manifestService = { ...mockService, manifest_managed: 1 }; diff --git a/client/src/components/pages/Services/ServiceForm.tsx b/client/src/components/pages/Services/ServiceForm.tsx index 80c3285..bc1fcfc 100644 --- a/client/src/components/pages/Services/ServiceForm.tsx +++ b/client/src/components/pages/Services/ServiceForm.tsx @@ -6,8 +6,11 @@ import type { CreateServiceInput, UpdateServiceInput, SchemaMapping, + MetricSchemaConfig, + HealthEndpointFormat, } from '../../../types/service'; import SchemaConfigEditor from './SchemaConfigEditor'; +import MetricSchemaConfigEditor from './MetricSchemaConfigEditor'; import styles from './ServiceForm.module.css'; interface ServiceFormProps { @@ -43,6 +46,26 @@ function parseSchemaConfig(raw: string | null): SchemaMapping | null { } } +function parseMetricSchemaConfig(raw: string | null): MetricSchemaConfig | null { + if (!raw) return null; + try { + const parsed = JSON.parse(raw); + if (parsed && (parsed.metrics || parsed.labels || parsed.latency_unit)) { + return parsed as MetricSchemaConfig; + } + return null; + } catch { + return null; + } +} + +const FORMAT_OPTIONS: { value: HealthEndpointFormat; label: string }[] = [ + { value: 'default', label: 'Default' }, + { value: 'schema', label: 'Custom Schema' }, + { value: 'prometheus', label: 'Prometheus' }, + { value: 'otlp', label: 'OTLP (Push)' }, +]; + function ServiceForm({ teams, service, onSuccess, onCancel }: ServiceFormProps) { const isEdit = !!service; @@ -52,10 +75,14 @@ function ServiceForm({ teams, service, onSuccess, onCancel }: ServiceFormProps) health_endpoint: service?.health_endpoint ?? '', metrics_endpoint: service?.metrics_endpoint ?? '', is_active: service?.is_active === 1, + health_endpoint_format: (service?.health_endpoint_format ?? 'default') as HealthEndpointFormat, }); const [schemaConfig, setSchemaConfig] = useState( parseSchemaConfig(service?.schema_config ?? null) ); + const [metricSchemaConfig, setMetricSchemaConfig] = useState( + parseMetricSchemaConfig(service?.schema_config ?? null) + ); const [errors, setErrors] = useState({}); const [submitError, setSubmitError] = useState(null); const [isSubmitting, setIsSubmitting] = useState(false); @@ -65,6 +92,14 @@ function ServiceForm({ teams, service, onSuccess, onCancel }: ServiceFormProps) setErrors((prev) => ({ ...prev, schema_config: undefined })); }, []); + const handleMetricSchemaChange = useCallback((value: MetricSchemaConfig | null) => { + setMetricSchemaConfig(value); + }, []); + + const requiresHealthEndpoint = formData.health_endpoint_format !== 'otlp'; + const showSchemaEditor = formData.health_endpoint_format === 'schema'; + const showMetricSchemaEditor = formData.health_endpoint_format === 'prometheus' || formData.health_endpoint_format === 'otlp'; + const validateForm = (): boolean => { const newErrors: FormErrors = {}; @@ -76,10 +111,12 @@ function ServiceForm({ teams, service, onSuccess, onCancel }: ServiceFormProps) newErrors.team_id = 'Team is required'; } - if (!formData.health_endpoint.trim()) { - newErrors.health_endpoint = 'Health endpoint is required'; - } else if (!isValidUrl(formData.health_endpoint)) { - newErrors.health_endpoint = 'Must be a valid HTTP or HTTPS URL'; + if (requiresHealthEndpoint) { + if (!formData.health_endpoint.trim()) { + newErrors.health_endpoint = 'Health endpoint is required'; + } else if (!isValidUrl(formData.health_endpoint)) { + newErrors.health_endpoint = 'Must be a valid HTTP or HTTPS URL'; + } } if (formData.metrics_endpoint && !isValidUrl(formData.metrics_endpoint)) { @@ -101,25 +138,32 @@ function ServiceForm({ teams, service, onSuccess, onCancel }: ServiceFormProps) setIsSubmitting(true); try { - const schemaConfigJson = schemaConfig ? JSON.stringify(schemaConfig) : null; + const schemaConfigJson = showSchemaEditor && schemaConfig + ? JSON.stringify(schemaConfig) + : showMetricSchemaEditor && metricSchemaConfig + ? JSON.stringify(metricSchemaConfig) + : null; + const healthEndpoint = requiresHealthEndpoint ? formData.health_endpoint : ''; if (isEdit && service) { const updateData: UpdateServiceInput = { name: formData.name, team_id: formData.team_id, - health_endpoint: formData.health_endpoint, + health_endpoint: healthEndpoint, metrics_endpoint: formData.metrics_endpoint || undefined, is_active: formData.is_active, schema_config: schemaConfigJson, + health_endpoint_format: formData.health_endpoint_format, }; await updateService(service.id, updateData); } else { const createData: CreateServiceInput = { name: formData.name, team_id: formData.team_id, - health_endpoint: formData.health_endpoint, + health_endpoint: healthEndpoint, metrics_endpoint: formData.metrics_endpoint || undefined, schema_config: schemaConfigJson, + health_endpoint_format: formData.health_endpoint_format, }; await createService(createData); } @@ -199,55 +243,98 @@ function ServiceForm({ teams, service, onSuccess, onCancel }: ServiceFormProps)
-
-
- - setFormData({ ...formData, metrics_endpoint: e.target.value })} - className={`${styles.input} ${errors.metrics_endpoint ? styles.inputError : ''}`} - placeholder="https://example.com/metrics" + {requiresHealthEndpoint && ( +
+ + setFormData({ ...formData, health_endpoint: e.target.value })} + className={`${styles.input} ${errors.health_endpoint ? styles.inputError : ''}`} + placeholder="https://example.com/dependencies" + disabled={isSubmitting} + aria-describedby={errors.health_endpoint ? 'health-endpoint-error' : undefined} + /> + {errors.health_endpoint && ( + + {errors.health_endpoint} + + )} + URL that returns dependency health status +
+ )} + + {requiresHealthEndpoint && ( +
+ + setFormData({ ...formData, metrics_endpoint: e.target.value })} + className={`${styles.input} ${errors.metrics_endpoint ? styles.inputError : ''}`} + placeholder="https://example.com/metrics" + disabled={isSubmitting} + aria-describedby={errors.metrics_endpoint ? 'metrics-endpoint-error' : undefined} + /> + {errors.metrics_endpoint && ( + + {errors.metrics_endpoint} + + )} + Optional URL for metrics data +
+ )} + + {showSchemaEditor && ( + - {errors.metrics_endpoint && ( - - {errors.metrics_endpoint} - - )} - Optional URL for metrics data -
+ )} - + {showMetricSchemaEditor && ( + + )} {isEdit && (
diff --git a/client/src/components/pages/Services/Services.module.css b/client/src/components/pages/Services/Services.module.css index 06e3e22..51a9b4f 100644 --- a/client/src/components/pages/Services/Services.module.css +++ b/client/src/components/pages/Services/Services.module.css @@ -653,6 +653,19 @@ margin-left: 0.5rem; } +.formatBadge { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: 0.75rem; + font-weight: 500; + color: var(--color-text-secondary); + background-color: var(--color-bg-hover); + border: 1px solid var(--color-border); + border-radius: 9999px; + margin-left: 0.5rem; +} + /* Responsive */ @media (max-width: 640px) { .container { diff --git a/client/src/components/pages/Teams/ApiKeys.module.css b/client/src/components/pages/Teams/ApiKeys.module.css new file mode 100644 index 0000000..cacd1d5 --- /dev/null +++ b/client/src/components/pages/Teams/ApiKeys.module.css @@ -0,0 +1,375 @@ +.container { + display: flex; + flex-direction: column; + gap: var(--space-4); +} + +.header { + display: flex; + justify-content: space-between; + align-items: flex-start; +} + +.title { + font-size: var(--font-lg); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +.subtitle { + font-size: var(--font-sm); + color: var(--color-text-muted); + margin: 0.25rem 0 0; +} + +.createButton { + display: inline-flex; + align-items: center; + gap: 0.375rem; + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text-inverse); + background-color: var(--color-accent); + border: 1px solid transparent; + border-radius: var(--radius-md); + cursor: pointer; + transition: background-color var(--duration-fast); +} + +.createButton:hover { + background-color: var(--color-accent-hover); +} + +/* Revealed key card */ +.revealedKeyCard { + padding: var(--space-4); + background-color: var(--color-success-bg, rgba(34, 197, 94, 0.08)); + border: 1px solid var(--color-success-border, rgba(34, 197, 94, 0.3)); + border-radius: var(--radius-md); +} + +.revealedKeyHeader { + display: flex; + align-items: center; + gap: 0.5rem; + color: var(--color-healthy); + margin-bottom: 0.5rem; +} + +.revealedKeyWarning { + font-size: var(--font-sm); + color: var(--color-text-muted); + margin: 0 0 0.75rem; +} + +.revealedKeyValue { + display: flex; + align-items: center; + gap: 0.5rem; + padding: 0.5rem 0.75rem; + background-color: var(--color-bg-input); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + margin-bottom: 0.75rem; + overflow-x: auto; +} + +.revealedKeyValue code { + font-size: var(--font-sm); + word-break: break-all; + flex: 1; +} + +.copyButton { + display: inline-flex; + align-items: center; + padding: 0.25rem; + color: var(--color-text-muted); + background: none; + border: none; + cursor: pointer; + border-radius: var(--radius-sm); + transition: color var(--duration-fast); + flex-shrink: 0; +} + +.copyButton:hover { + color: var(--color-text); +} + +.dismissButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); + background-color: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + cursor: pointer; +} + +.dismissButton:hover { + background-color: var(--color-bg-hover); +} + +/* Create form */ +.createForm { + display: flex; + flex-direction: column; + gap: 0.75rem; + padding: var(--space-3); + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); +} + +.nameInput { + padding: 0.5rem 0.75rem; + font-size: var(--font-sm); + border: 1px solid var(--color-border-input); + border-radius: var(--radius-md); + background-color: var(--color-bg-input); + color: var(--color-text); +} + +.nameInput:focus { + outline: none; + border-color: var(--color-accent); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +.createActions { + display: flex; + gap: 0.5rem; + justify-content: flex-end; +} + +.cancelButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); + background-color: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + cursor: pointer; +} + +.cancelButton:hover:not(:disabled) { + background-color: var(--color-bg-hover); +} + +.generateButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text-inverse); + background-color: var(--color-accent); + border: 1px solid transparent; + border-radius: var(--radius-md); + cursor: pointer; +} + +.generateButton:hover:not(:disabled) { + background-color: var(--color-accent-hover); +} + +.generateButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +/* Empty state */ +.emptyState { + display: flex; + flex-direction: column; + align-items: center; + padding: var(--space-6); + color: var(--color-text-muted); + text-align: center; +} + +.emptyState p { + margin: 0.5rem 0 0; +} + +.emptyIcon { + opacity: 0.4; +} + +.emptyHint { + font-size: var(--font-sm); +} + +/* Key list */ +.keyList { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.keyItem { + display: flex; + align-items: center; + gap: var(--space-3); + padding: 0.75rem 1rem; + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); +} + +.keyInfo { + display: flex; + flex-direction: column; + gap: 0.125rem; + flex: 1; + min-width: 0; +} + +.keyName { + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); +} + +.keyPrefix { + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +/* Rate limit column */ +.rateLimit { + display: flex; + align-items: center; + gap: 0.375rem; + flex-shrink: 0; +} + +.rateLimitValue { + font-size: var(--font-sm); + color: var(--color-text); + font-variant-numeric: tabular-nums; +} + +.rateLimitLabel { + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.lockIcon { + color: var(--color-text-muted); + flex-shrink: 0; +} + +.editButton { + display: inline-flex; + align-items: center; + padding: 0.25rem; + color: var(--color-text-muted); + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast), border-color var(--duration-fast); + flex-shrink: 0; +} + +.editButton:hover { + color: var(--color-accent); + border-color: var(--color-accent); +} + +/* Rate limit edit dialog */ +.rateLimitDialog { + display: flex; + flex-direction: column; + gap: 0.75rem; +} + +.rateLimitDialogDesc { + font-size: var(--font-sm); + color: var(--color-text-secondary); + margin: 0; + line-height: 1.5; +} + +.rateLimitDialogLabel { + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); +} + +.rateLimitDialogError { + font-size: var(--font-xs); + color: var(--color-critical); + margin: -0.25rem 0 0; +} + +.rateLimitDialogActions { + display: flex; + justify-content: space-between; + align-items: center; + margin-top: 0.25rem; +} + +.rateLimitDialogRight { + display: flex; + gap: 0.5rem; +} + +.keyMeta { + display: flex; + flex-direction: column; + gap: 0.125rem; + font-size: var(--font-xs); + color: var(--color-text-muted); + text-align: right; + flex-shrink: 0; +} + +.deleteButton { + display: inline-flex; + align-items: center; + padding: 0.375rem; + color: var(--color-text-muted); + background: none; + border: 1px solid transparent; + border-radius: var(--radius-md); + cursor: pointer; + transition: color var(--duration-fast), border-color var(--duration-fast); + flex-shrink: 0; +} + +.deleteButton:hover { + color: var(--color-critical); + border-color: var(--color-critical); +} + +/* Help section */ +.helpSection { + padding: var(--space-3); + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); +} + +.helpTitle { + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); + margin: 0 0 0.5rem; +} + +.codeBlock { + padding: 0.75rem 1rem; + font-size: var(--font-xs); + line-height: 1.6; + color: var(--color-text-secondary); + background-color: var(--color-bg-input); + border: 1px solid var(--color-border); + border-radius: var(--radius-sm); + overflow-x: auto; + margin: 0; +} diff --git a/client/src/components/pages/Teams/ApiKeys.test.tsx b/client/src/components/pages/Teams/ApiKeys.test.tsx new file mode 100644 index 0000000..f1b241d --- /dev/null +++ b/client/src/components/pages/Teams/ApiKeys.test.tsx @@ -0,0 +1,379 @@ +import { render, screen, fireEvent, waitFor } from '@testing-library/react'; +import ApiKeys from './ApiKeys'; + +// Mock HTMLDialogElement for ConfirmDialog +beforeAll(() => { + HTMLDialogElement.prototype.showModal = jest.fn(); + HTMLDialogElement.prototype.close = jest.fn(); +}); + +const mockFetch = jest.fn(); +global.fetch = mockFetch; + +function jsonResponse(data: unknown, status = 200) { + return { + ok: status >= 200 && status < 300, + status, + json: () => Promise.resolve(data), + }; +} + +const mockKeys = [ + { + id: 'k1', + team_id: 't1', + name: 'Production Collector', + key_prefix: 'dps_a1b2c3d4', + last_used_at: '2026-03-14T10:00:00Z', + created_at: '2026-03-01T10:00:00Z', + created_by: 'u1', + }, + { + id: 'k2', + team_id: 't1', + name: 'Staging Collector', + key_prefix: 'dps_e5f6g7h8', + last_used_at: null, + created_at: '2026-03-10T10:00:00Z', + created_by: 'u1', + }, +]; + +beforeEach(() => { + mockFetch.mockReset(); +}); + +describe('ApiKeys', () => { + it('renders key list with prefix and dates', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(mockKeys)); + + render(); + + await waitFor(() => { + expect(screen.getByText('Production Collector')).toBeInTheDocument(); + expect(screen.getByText('Staging Collector')).toBeInTheDocument(); + }); + + expect(screen.getByText('dps_a1b2c3d4...')).toBeInTheDocument(); + expect(screen.getByText('dps_e5f6g7h8...')).toBeInTheDocument(); + expect(screen.getByText('Never used')).toBeInTheDocument(); + }); + + it('renders empty state correctly', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([])); + + render(); + + await waitFor(() => { + expect(screen.getByText('No API keys yet.')).toBeInTheDocument(); + }); + + expect(screen.getByText(/Create a key to start pushing OTLP metrics/)).toBeInTheDocument(); + }); + + it('empty state does not show create hint for non-managers', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([])); + + render(); + + await waitFor(() => { + expect(screen.getByText('No API keys yet.')).toBeInTheDocument(); + }); + + expect(screen.queryByText(/Create a key to start pushing OTLP metrics/)).not.toBeInTheDocument(); + }); + + it('create shows raw key once with copy button', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(mockKeys)); // initial load + mockFetch.mockResolvedValueOnce( + jsonResponse({ + id: 'k3', + team_id: 't1', + name: 'New Key', + key_prefix: 'dps_newkey12', + rawKey: 'dps_newkey1234567890abcdef1234567890abcdef', + last_used_at: null, + created_at: '2026-03-15T10:00:00Z', + created_by: 'u1', + }, 201) + ); + mockFetch.mockResolvedValueOnce(jsonResponse([...mockKeys, { id: 'k3', team_id: 't1', name: 'New Key', key_prefix: 'dps_newkey12', last_used_at: null, created_at: '2026-03-15T10:00:00Z', created_by: 'u1' }])); // reload + + render(); + + await waitFor(() => { + expect(screen.getByText('Production Collector')).toBeInTheDocument(); + }); + + // Click create button + fireEvent.click(screen.getByText('Create Key')); + + // Fill name and generate + fireEvent.change(screen.getByPlaceholderText(/Key name/), { target: { value: 'New Key' } }); + fireEvent.click(screen.getByText('Generate')); + + await waitFor(() => { + expect(screen.getByText('API Key Created')).toBeInTheDocument(); + }); + + expect(screen.getByText('dps_newkey1234567890abcdef1234567890abcdef')).toBeInTheDocument(); + expect(screen.getByText(/will not be shown again/)).toBeInTheDocument(); + }); + + it('delete shows confirm dialog', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(mockKeys)); + + render(); + + await waitFor(() => { + expect(screen.getByText('Production Collector')).toBeInTheDocument(); + }); + + // Click delete button on first key + const deleteButtons = screen.getAllByTitle('Revoke key'); + fireEvent.click(deleteButtons[0]); + + // Confirm dialog should appear + expect(screen.getByText('Revoke API Key')).toBeInTheDocument(); + expect(screen.getByText(/Any collectors using it will no longer be able to push metrics/)).toBeInTheDocument(); + }); + + it('non-manager cannot see Create Key button', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(mockKeys)); + + render(); + + await waitFor(() => { + expect(screen.getByText('Production Collector')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Create Key')).not.toBeInTheDocument(); + }); + + it('non-manager cannot see delete buttons', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(mockKeys)); + + render(); + + await waitFor(() => { + expect(screen.getByText('Production Collector')).toBeInTheDocument(); + }); + + expect(screen.queryByTitle('Revoke key')).not.toBeInTheDocument(); + }); + + it('shows collector configuration help text', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([])); + + render(); + + await waitFor(() => { + expect(screen.getByText('Collector Configuration')).toBeInTheDocument(); + }); + + expect(screen.getByText(/otlphttp/)).toBeInTheDocument(); + expect(screen.getByText(/Bearer dps_/)).toBeInTheDocument(); + }); + + it('handles API error when loading keys', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse({ message: 'Unauthorized' }, 401)); + + render(); + + await waitFor(() => { + expect(screen.getByText('Unauthorized')).toBeInTheDocument(); + }); + }); + + it('cancel button hides create form', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([])); + + render(); + + await waitFor(() => { + expect(screen.getByText('Create Key')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByText('Create Key')); + expect(screen.getByPlaceholderText(/Key name/)).toBeInTheDocument(); + + fireEvent.click(screen.getByText('Cancel')); + expect(screen.queryByPlaceholderText(/Key name/)).not.toBeInTheDocument(); + }); + + it('generate button is disabled when name is empty', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([])); + + render(); + + await waitFor(() => { + expect(screen.getByText('Create Key')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByText('Create Key')); + + const generateButton = screen.getByText('Generate'); + expect(generateButton).toBeDisabled(); + }); + + // --- Rate limit column tests (DPS-102b/c/d) --- + + it('displays (default) when rate_limit_rpm is null', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: null, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByText('(default)')).toBeInTheDocument(); + }); + }); + + it('displays (custom) when rate_limit_rpm is set', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: 50000, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByText('(custom)')).toBeInTheDocument(); + }); + }); + + it('renders lock icon when rate_limit_admin_locked is 1', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: 50000, rate_limit_admin_locked: 1 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Locked by admin')).toBeInTheDocument(); + }); + }); + + it('does not render edit icon when rate_limit_admin_locked is 1', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: 50000, rate_limit_admin_locked: 1 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Locked by admin')).toBeInTheDocument(); + }); + + expect(screen.queryByTitle('Edit rate limit')).not.toBeInTheDocument(); + }); + + it('renders edit icon when canManage and key is not locked', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: null, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + }); + + it('does not render edit icon when canManage is false', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: null, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByText('Production Collector')).toBeInTheDocument(); + }); + + expect(screen.queryByTitle('Edit rate limit')).not.toBeInTheDocument(); + }); + + it('rate limit edit dialog validates and saves correctly', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: null, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + expect(screen.getByText('Edit Rate Limit')).toBeInTheDocument(); + + const input = screen.getByPlaceholderText('150000'); + fireEvent.change(input, { target: { value: '80000' } }); + + // Mock PATCH and reload + mockFetch.mockResolvedValueOnce(jsonResponse({ ok: true })); + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: 80000, rate_limit_admin_locked: 0 }, + ])); + + fireEvent.click(screen.getByText('Save')); + + await waitFor(() => { + const patchCall = mockFetch.mock.calls.find( + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (c: any[]) => c[1]?.method === 'PATCH' + ); + expect(patchCall).toBeDefined(); + expect(JSON.parse(patchCall![1].body)).toEqual({ rate_limit_rpm: 80000 }); + }); + }); + + it('Reset to default sends null in rate limit edit dialog', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: 50000, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + // Mock PATCH and reload + mockFetch.mockResolvedValueOnce(jsonResponse({ ok: true })); + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: null, rate_limit_admin_locked: 0 }, + ])); + + fireEvent.click(screen.getByText('Reset to default')); + + await waitFor(() => { + const patchCall = mockFetch.mock.calls.find( + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (c: any[]) => c[1]?.method === 'PATCH' + ); + expect(patchCall).toBeDefined(); + expect(JSON.parse(patchCall![1].body)).toEqual({ rate_limit_rpm: null }); + }); + }); + + it('displays Unlimited with (admin) when rate_limit_rpm is 0', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse([ + { ...mockKeys[0], rate_limit_rpm: 0, rate_limit_admin_locked: 0 }, + ])); + + render(); + + await waitFor(() => { + expect(screen.getByText('Unlimited')).toBeInTheDocument(); + expect(screen.getByText('(admin)')).toBeInTheDocument(); + }); + + // Edit button should not show for unlimited keys (rpm=0) + expect(screen.queryByTitle('Edit rate limit')).not.toBeInTheDocument(); + }); +}); diff --git a/client/src/components/pages/Teams/ApiKeys.tsx b/client/src/components/pages/Teams/ApiKeys.tsx new file mode 100644 index 0000000..8332bd0 --- /dev/null +++ b/client/src/components/pages/Teams/ApiKeys.tsx @@ -0,0 +1,407 @@ +import { useState, useEffect, useCallback } from 'react'; +import { Key, Trash2, Copy, Check, Plus, Lock, Pencil } from 'lucide-react'; +import { listApiKeys, createApiKey, deleteApiKey } from '../../../api/apiKeys'; +import type { ApiKey } from '../../../api/apiKeys'; +import { updateApiKeyRateLimit } from '../../../api/otlpStats'; +import { formatRelativeTime } from '../../../utils/formatting'; +import ConfirmDialog from '../../common/ConfirmDialog'; +import Modal from '../../common/Modal'; +import styles from './Teams.module.css'; +import apiKeyStyles from './ApiKeys.module.css'; + +const DEFAULT_RATE_LIMIT_RPM = 150_000; + +interface ApiKeysProps { + teamId: string; + canManage: boolean; +} + +function ApiKeys({ teamId, canManage }: ApiKeysProps) { + const [keys, setKeys] = useState([]); + const [isLoading, setIsLoading] = useState(true); + const [error, setError] = useState(null); + const [newKeyName, setNewKeyName] = useState(''); + const [isCreating, setIsCreating] = useState(false); + const [showCreateForm, setShowCreateForm] = useState(false); + const [revealedKey, setRevealedKey] = useState(null); + const [copied, setCopied] = useState(false); + const [deleteKeyId, setDeleteKeyId] = useState(null); + const [isDeleting, setIsDeleting] = useState(false); + + // Rate limit edit dialog state + const [editRateLimitKey, setEditRateLimitKey] = useState(null); + const [rateLimitInput, setRateLimitInput] = useState(''); + const [rateLimitError, setRateLimitError] = useState(null); + const [isSavingRateLimit, setIsSavingRateLimit] = useState(false); + + const loadKeys = useCallback(async () => { + try { + setIsLoading(true); + const result = await listApiKeys(teamId); + setKeys(result); + setError(null); + } catch (err) { + setError(err instanceof Error ? err.message : 'Failed to load API keys'); + } finally { + setIsLoading(false); + } + }, [teamId]); + + useEffect(() => { + loadKeys(); + }, [loadKeys]); + + const handleCreate = async () => { + if (!newKeyName.trim()) return; + try { + setIsCreating(true); + const result = await createApiKey(teamId, newKeyName.trim()); + setRevealedKey(result.rawKey); + setNewKeyName(''); + setShowCreateForm(false); + await loadKeys(); + } catch (err) { + setError(err instanceof Error ? err.message : 'Failed to create API key'); + } finally { + setIsCreating(false); + } + }; + + const handleDelete = async () => { + if (!deleteKeyId) return; + try { + setIsDeleting(true); + await deleteApiKey(teamId, deleteKeyId); + setDeleteKeyId(null); + await loadKeys(); + } catch (err) { + setError(err instanceof Error ? err.message : 'Failed to delete API key'); + } finally { + setIsDeleting(false); + } + }; + + const handleCopy = async (text: string) => { + try { + await navigator.clipboard.writeText(text); + setCopied(true); + setTimeout(() => setCopied(false), 2000); + } catch { + // Fallback: select the text + } + }; + + const dismissRevealedKey = () => { + setRevealedKey(null); + }; + + const openRateLimitDialog = (key: ApiKey) => { + setEditRateLimitKey(key); + setRateLimitInput(key.rate_limit_rpm !== null ? String(key.rate_limit_rpm) : ''); + setRateLimitError(null); + }; + + const closeRateLimitDialog = () => { + setEditRateLimitKey(null); + setRateLimitInput(''); + setRateLimitError(null); + }; + + const validateRateLimitInput = (value: string): string | null => { + if (value === '') return null; // empty = will reset to default + const num = Number(value); + if (!Number.isInteger(num) || num <= 0) return 'Must be a positive integer'; + if (num > 1_500_000) return 'Cannot exceed 1,500,000 req/min'; + return null; + }; + + const handleSaveRateLimit = async () => { + if (!editRateLimitKey) return; + const trimmed = rateLimitInput.trim(); + const validationError = validateRateLimitInput(trimmed); + if (validationError) { + setRateLimitError(validationError); + return; + } + const newLimit = trimmed === '' ? null : Number(trimmed); + try { + setIsSavingRateLimit(true); + await updateApiKeyRateLimit(teamId, editRateLimitKey.id, newLimit); + closeRateLimitDialog(); + await loadKeys(); + } catch (err) { + setRateLimitError(err instanceof Error ? err.message : 'Failed to update rate limit'); + } finally { + setIsSavingRateLimit(false); + } + }; + + const handleResetToDefault = async () => { + if (!editRateLimitKey) return; + try { + setIsSavingRateLimit(true); + await updateApiKeyRateLimit(teamId, editRateLimitKey.id, null); + closeRateLimitDialog(); + await loadKeys(); + } catch (err) { + setRateLimitError(err instanceof Error ? err.message : 'Failed to reset rate limit'); + } finally { + setIsSavingRateLimit(false); + } + }; + + const getEffectiveRateLimit = (key: ApiKey): number => { + return key.rate_limit_rpm ?? DEFAULT_RATE_LIMIT_RPM; + }; + + const isCustomRateLimit = (key: ApiKey): boolean => { + return key.rate_limit_rpm !== null; + }; + + const isAdminLocked = (key: ApiKey): boolean => { + return key.rate_limit_admin_locked === 1; + }; + + if (isLoading) { + return ( +
+
+
+ Loading API keys... +
+
+ ); + } + + return ( +
+
+
+

API Keys

+

+ Authenticate OTLP metric pushes from your collectors. +

+
+ {canManage && !showCreateForm && !revealedKey && ( + + )} +
+ + {error && ( +
+ {error} +
+ )} + + {revealedKey && ( +
+
+ + API Key Created +
+

+ Copy this key now. It will not be shown again. +

+
+ {revealedKey} + +
+ +
+ )} + + {canManage && showCreateForm && ( +
+ setNewKeyName(e.target.value)} + placeholder="Key name (e.g., Production Collector)" + className={apiKeyStyles.nameInput} + disabled={isCreating} + onKeyDown={(e) => { + if (e.key === 'Enter') handleCreate(); + }} + /> +
+ + +
+
+ )} + + {keys.length === 0 ? ( +
+ +

No API keys yet.

+ {canManage &&

Create a key to start pushing OTLP metrics.

} +
+ ) : ( +
+ {keys.map((key) => ( +
+
+ {key.name} + {key.key_prefix}... +
+
+ + {key.rate_limit_rpm === 0 + ? 'Unlimited' + : `${getEffectiveRateLimit(key).toLocaleString()} req/min`} + + + {key.rate_limit_rpm === 0 + ? '(admin)' + : isCustomRateLimit(key) ? '(custom)' : '(default)'} + + {isAdminLocked(key) && ( + + + + )} + {canManage && !isAdminLocked(key) && key.rate_limit_rpm !== 0 && ( + + )} +
+
+ Created {formatRelativeTime(key.created_at)} + {key.last_used_at ? `Last used ${formatRelativeTime(key.last_used_at)}` : 'Never used'} +
+ {canManage && ( + + )} +
+ ))} +
+ )} + +
+

Collector Configuration

+ 
+{`exporters:
+  otlphttp:
+    endpoint: "https:///v1/metrics"
+    headers:
+      Authorization: "Bearer dps_..."`}
+
+
+ + setDeleteKeyId(null)} + onConfirm={handleDelete} + title="Revoke API Key" + message="Are you sure you want to revoke this API key? Any collectors using it will no longer be able to push metrics." + confirmLabel="Revoke" + isDestructive + isLoading={isDeleting} + /> + + + {editRateLimitKey && ( +
+

+ Set the rate limit for {editRateLimitKey.name} ({editRateLimitKey.key_prefix}...). + Leave empty to use the system default ({DEFAULT_RATE_LIMIT_RPM.toLocaleString()} req/min). +

+ + { + setRateLimitInput(e.target.value); + setRateLimitError(null); + }} + placeholder={String(DEFAULT_RATE_LIMIT_RPM)} + className={apiKeyStyles.nameInput} + min={1} + disabled={isSavingRateLimit} + onKeyDown={(e) => { + if (e.key === 'Enter') handleSaveRateLimit(); + }} + /> + {rateLimitError && ( +

{rateLimitError}

+ )} +
+ +
+ + +
+
+
+ )} + +
+ ); +} + +export default ApiKeys; diff --git a/client/src/components/pages/Teams/OtlpStats.module.css b/client/src/components/pages/Teams/OtlpStats.module.css new file mode 100644 index 0000000..7ff2c39 --- /dev/null +++ b/client/src/components/pages/Teams/OtlpStats.module.css @@ -0,0 +1,437 @@ +.container { + display: flex; + flex-direction: column; + gap: var(--space-4); +} + +.header { + display: flex; + justify-content: space-between; + align-items: flex-start; +} + +.title { + font-size: var(--font-lg); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +.subtitle { + font-size: var(--font-sm); + color: var(--color-text-muted); + margin: 0.25rem 0 0; +} + +.summaryGrid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(10rem, 1fr)); + gap: var(--space-3); +} + +.summaryCard { + padding: var(--space-3); + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + text-align: center; +} + +.summaryValue { + font-size: var(--font-2xl); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +.summaryLabel { + font-size: var(--font-xs); + color: var(--color-text-muted); + margin: 0.25rem 0 0; +} + +.summaryCardWarning { + composes: summaryCard; + border-color: var(--color-warning-border, rgba(234, 179, 8, 0.3)); + background-color: var(--color-warning-bg, rgba(234, 179, 8, 0.06)); +} + +.summaryCardError { + composes: summaryCard; + border-color: var(--color-critical-border, rgba(239, 68, 68, 0.3)); + background-color: var(--color-critical-bg, rgba(239, 68, 68, 0.06)); +} + +.sectionTitle { + font-size: var(--font-base); + font-weight: var(--font-semibold); + color: var(--color-text); + margin: 0; +} + +.tableWrapper { + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + overflow-x: auto; +} + +.table { + width: 100%; + border-collapse: collapse; + font-size: var(--font-sm); +} + +.table th { + text-align: left; + padding: 0.5rem 0.75rem; + font-weight: 500; + color: var(--color-text-muted); + border-bottom: 1px solid var(--color-border); + background-color: var(--color-surface); + white-space: nowrap; +} + +.table td { + padding: 0.5rem 0.75rem; + color: var(--color-text); + border-bottom: 1px solid var(--color-border); +} + +.table tbody tr:last-child td { + border-bottom: none; +} + +.table tbody tr:hover { + background-color: var(--color-bg-hover); +} + +.badge { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: var(--font-xs); + font-weight: 500; + border-radius: 9999px; + white-space: nowrap; +} + +.badgeSuccess { + composes: badge; + color: var(--color-healthy); + background-color: var(--color-success-bg, rgba(34, 197, 94, 0.08)); +} + +.badgeError { + composes: badge; + color: var(--color-critical); + background-color: var(--color-critical-bg, rgba(239, 68, 68, 0.06)); +} + +.badgeNeutral { + composes: badge; + color: var(--color-text-muted); + background-color: var(--color-surface); +} + +.badgeInactive { + composes: badge; + color: var(--color-text-muted); + background-color: var(--color-bg-hover); +} + +.warningsList { + margin: 0; + padding: 0 0 0 1rem; + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.warningsList li { + margin: 0.125rem 0; +} + +.errorText { + font-size: var(--font-xs); + color: var(--color-critical); + max-width: 20rem; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.errorText:hover { + white-space: normal; + overflow: visible; +} + +.emptyState { + display: flex; + flex-direction: column; + align-items: center; + padding: var(--space-6); + color: var(--color-text-muted); + text-align: center; +} + +.emptyState p { + margin: 0.5rem 0 0; +} + +.emptyIcon { + opacity: 0.4; +} + +.keyList { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.keyCard { + display: flex; + flex-direction: column; + gap: 0.5rem; + padding: 0.75rem 1rem; + background-color: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + font-size: var(--font-sm); +} + +.keyCardHeader { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: var(--space-3); +} + +.keyCardInfo { + display: flex; + flex-direction: column; + gap: 0.125rem; + min-width: 0; +} + +.keyName { + display: flex; + align-items: center; + gap: 0.5rem; + font-weight: 500; + color: var(--color-text); +} + +.keyPrefix { + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.keyMeta { + font-size: var(--font-xs); + color: var(--color-text-muted); + text-align: right; + flex-shrink: 0; +} + +/* Warning badges */ +.badgeWarning { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: var(--font-xs); + font-weight: 500; + border-radius: 9999px; + white-space: nowrap; + color: var(--color-warning, #d97706); + background-color: var(--color-warning-bg, rgba(234, 179, 8, 0.08)); +} + +.badgeMutedWarning { + display: inline-flex; + align-items: center; + padding: 0.125rem 0.5rem; + font-size: var(--font-xs); + font-weight: 500; + border-radius: 9999px; + white-space: nowrap; + color: var(--color-text-muted); + background-color: var(--color-bg-hover); +} + +/* Usage summary row */ +.usageSummary { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 0.5rem; + font-size: var(--font-xs); + color: var(--color-text-muted); +} + +.rejectedWarning { + color: var(--color-warning, #d97706); + font-weight: 500; +} + +/* Rate limit row */ +.rateLimitRow { + display: flex; + align-items: center; + gap: 0.5rem; + font-size: var(--font-xs); +} + +.rateLimitText { + color: var(--color-text-muted); +} + +.rateLimitSuffix { + color: var(--color-text-muted); + opacity: 0.7; +} + +.lockIcon { + color: var(--color-text-muted); + flex-shrink: 0; +} + +.editButton { + display: inline-flex; + align-items: center; + padding: 0.25rem; + color: var(--color-text-muted); + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast), border-color var(--duration-fast); + flex-shrink: 0; +} + +.editButton:hover { + color: var(--color-accent); + border-color: var(--color-accent); +} + +.expandButton { + display: inline-flex; + align-items: center; + gap: 0.25rem; + margin-left: auto; + padding: 0.25rem 0.5rem; + font-size: var(--font-xs); + color: var(--color-text-muted); + background: none; + border: 1px solid var(--color-border); + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast), border-color var(--duration-fast); +} + +.expandButton:hover { + color: var(--color-accent); + border-color: var(--color-accent); +} + +/* Chart container */ +.chartContainer { + border-top: 1px solid var(--color-border); + padding-top: 0.5rem; +} + +/* Rate limit edit dialog */ +.rateLimitDialog { + display: flex; + flex-direction: column; + gap: 0.75rem; +} + +.rateLimitDialogDesc { + font-size: var(--font-sm); + color: var(--color-text-secondary); + margin: 0; + line-height: 1.5; +} + +.rateLimitDialogLabel { + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); +} + +.rateLimitDialogInput { + padding: 0.5rem 0.75rem; + font-size: var(--font-sm); + border: 1px solid var(--color-border-input); + border-radius: var(--radius-md); + background-color: var(--color-bg-input); + color: var(--color-text); +} + +.rateLimitDialogInput:focus { + outline: none; + border-color: var(--color-accent); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.1); +} + +.rateLimitDialogError { + font-size: var(--font-xs); + color: var(--color-critical); + margin: -0.25rem 0 0; +} + +.rateLimitDialogActions { + display: flex; + justify-content: space-between; + align-items: center; + margin-top: 0.25rem; +} + +.rateLimitDialogRight { + display: flex; + gap: 0.5rem; +} + +.dialogSecondaryButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text); + background-color: var(--color-bg-card); + border: 1px solid var(--color-border); + border-radius: var(--radius-md); + cursor: pointer; +} + +.dialogSecondaryButton:hover:not(:disabled) { + background-color: var(--color-bg-hover); +} + +.dialogSecondaryButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.dialogPrimaryButton { + padding: 0.375rem 0.75rem; + font-size: var(--font-sm); + font-weight: 500; + color: var(--color-text-inverse); + background-color: var(--color-accent); + border: 1px solid transparent; + border-radius: var(--radius-md); + cursor: pointer; +} + +.dialogPrimaryButton:hover:not(:disabled) { + background-color: var(--color-accent-hover); +} + +.dialogPrimaryButton:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.monoText { + font-family: var(--font-mono, monospace); + font-size: var(--font-xs); +} diff --git a/client/src/components/pages/Teams/OtlpStats.test.tsx b/client/src/components/pages/Teams/OtlpStats.test.tsx new file mode 100644 index 0000000..75939fa --- /dev/null +++ b/client/src/components/pages/Teams/OtlpStats.test.tsx @@ -0,0 +1,366 @@ +import { render, screen, fireEvent, waitFor } from '@testing-library/react'; +import OtlpStats from './OtlpStats'; + +beforeAll(() => { + HTMLDialogElement.prototype.showModal = jest.fn(); + HTMLDialogElement.prototype.close = jest.fn(); +}); + +const mockFetch = jest.fn(); +global.fetch = mockFetch; + +function jsonResponse(data: unknown, status = 200) { + return { + ok: status >= 200 && status < 300, + status, + json: () => Promise.resolve(data), + }; +} + +const baseKey = { + id: 'k1', + name: 'Prod Key', + key_prefix: 'dps_abc123', + last_used_at: '2026-04-04T10:00:00Z', + created_at: '2026-03-01T10:00:00Z', + rate_limit_rpm: 150000, + rate_limit_is_custom: false, + rate_limit_admin_locked: false, + usage_1h: 500, + usage_24h: 8000, + usage_7d: 50000, + rejected_24h: 0, + rejected_7d: 0, +}; + +function makeStatsResponse(keyOverrides = {}) { + return { + services: [ + { + id: 's1', + name: 'my-service', + is_active: 1, + last_push_success: 1, + last_push_error: null, + last_push_warnings: null, + last_push_at: '2026-04-04T09:00:00Z', + dependency_count: 3, + errors_24h: 0, + schema_config: null, + }, + ], + apiKeys: [{ ...baseKey, ...keyOverrides }], + summary: { + total_otlp_services: 1, + active_services: 1, + services_with_errors: 0, + services_never_pushed: 0, + }, + }; +} + +beforeEach(() => { + mockFetch.mockReset(); + localStorage.clear(); +}); + +describe('OtlpStats', () => { + // --- Warning badge tests (DPS-102d) --- + + it('renders warning badge when rejected_24h > 0', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ rejected_24h: 42 }))); + + render(); + + await waitFor(() => { + expect(screen.getByText('Approaching limit')).toBeInTheDocument(); + }); + }); + + it('does not render warning badge when rejected_24h is 0', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ rejected_24h: 0, rejected_7d: 0 }))); + + render(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Approaching limit')).not.toBeInTheDocument(); + }); + + it('renders muted 7d rejection text when rejected_7d > 0 but rejected_24h is 0', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ rejected_24h: 0, rejected_7d: 15 }))); + + render(); + + await waitFor(() => { + expect(screen.getByText('15 rejected in 7d')).toBeInTheDocument(); + }); + + expect(screen.queryByText('Approaching limit')).not.toBeInTheDocument(); + }); + + it('renders rejected_24h count in usage summary', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ rejected_24h: 10 }))); + + render(); + + await waitFor(() => { + expect(screen.getByText('10 rejected in 24h')).toBeInTheDocument(); + }); + }); + + // --- Rate limit display tests --- + + it('renders edit button for team lead on unlocked key', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + }); + + it('does not render edit button when rate_limit_admin_locked is true', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ rate_limit_admin_locked: true }))); + + render(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + expect(screen.queryByTitle('Edit rate limit')).not.toBeInTheDocument(); + }); + + it('renders lock icon when rate_limit_admin_locked is true', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ rate_limit_admin_locked: true }))); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Locked by admin')).toBeInTheDocument(); + }); + }); + + it('does not render edit button when canManage is false', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + expect(screen.queryByTitle('Edit rate limit')).not.toBeInTheDocument(); + }); + + it('displays (default) suffix for non-custom rate limit', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByText('(default)')).toBeInTheDocument(); + }); + }); + + it('displays (custom) suffix for custom rate limit', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ + rate_limit_rpm: 50000, + rate_limit_is_custom: true, + }))); + + render(); + + await waitFor(() => { + expect(screen.getByText('(custom)')).toBeInTheDocument(); + }); + }); + + // --- Rate limit edit dialog tests (DPS-102b) --- + + it('opens rate limit dialog on pencil click', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + expect(screen.getByText('Edit Rate Limit')).toBeInTheDocument(); + expect(screen.getByText(/Set the rate limit for/)).toBeInTheDocument(); + }); + + it('Save button is disabled when input is non-integer', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + const input = screen.getByPlaceholderText('150000'); + fireEvent.change(input, { target: { value: '-5' } }); + + expect(screen.getByText('Save')).toBeDisabled(); + }); + + it('Save button is enabled when input is empty (reset to default)', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + // Empty input = reset to default, should be valid + const input = screen.getByPlaceholderText('150000'); + fireEvent.change(input, { target: { value: '' } }); + + expect(screen.getByText('Save')).not.toBeDisabled(); + }); + + it('Reset to default calls updateApiKeyRateLimit with null', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(makeStatsResponse({ + rate_limit_rpm: 50000, + rate_limit_is_custom: true, + }))); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + // Mock the PATCH call and the stats reload + mockFetch.mockResolvedValueOnce(jsonResponse({ ok: true })); + mockFetch.mockResolvedValueOnce(jsonResponse(makeStatsResponse())); + + fireEvent.click(screen.getByText('Reset to default')); + + await waitFor(() => { + // Verify PATCH was called with null rate_limit_rpm + const patchCall = mockFetch.mock.calls.find( + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (c: any[]) => c[1]?.method === 'PATCH' + ); + expect(patchCall).toBeDefined(); + expect(JSON.parse(patchCall![1].body)).toEqual({ rate_limit_rpm: null }); + }); + }); + + it('Reset to default is disabled when key uses default rate limit', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse({ + rate_limit_is_custom: false, + }))); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + expect(screen.getByText('Reset to default')).toBeDisabled(); + }); + + it('saving a valid value calls updateApiKeyRateLimit with correct integer', async () => { + mockFetch.mockResolvedValueOnce(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByTitle('Edit rate limit')).toBeInTheDocument(); + }); + + fireEvent.click(screen.getByTitle('Edit rate limit')); + + const input = screen.getByPlaceholderText('150000'); + fireEvent.change(input, { target: { value: '75000' } }); + + // Mock the PATCH call and reload + mockFetch.mockResolvedValueOnce(jsonResponse({ ok: true })); + mockFetch.mockResolvedValueOnce(jsonResponse(makeStatsResponse())); + + fireEvent.click(screen.getByText('Save')); + + await waitFor(() => { + const patchCall = mockFetch.mock.calls.find( + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (c: any[]) => c[1]?.method === 'PATCH' + ); + expect(patchCall).toBeDefined(); + expect(JSON.parse(patchCall![1].body)).toEqual({ rate_limit_rpm: 75000 }); + }); + }); + + // --- Expand/collapse chart tests --- + + it('chart is not present before View usage button is clicked', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + // The chart title should not be in the DOM + expect(screen.queryByText('Prod Key (dps_abc123) — Usage')).not.toBeInTheDocument(); + }); + + it('View usage button mounts ApiKeyUsageChart on click', async () => { + // Initial stats load + mockFetch.mockResolvedValueOnce(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByText('Prod Key')).toBeInTheDocument(); + }); + + // Mock the chart's API call + mockFetch.mockResolvedValue(jsonResponse({ + api_key_id: 'k1', + granularity: 'minute', + from: '2026-04-03T10:00:00Z', + to: '2026-04-04T10:00:00Z', + buckets: [], + })); + + fireEvent.click(screen.getByTitle('View usage graph')); + + await waitFor(() => { + expect(screen.getByText('Prod Key (dps_abc123) — Usage')).toBeInTheDocument(); + }); + }); + + // --- Usage summary display --- + + it('renders usage summary row with push counts', async () => { + mockFetch.mockResolvedValue(jsonResponse(makeStatsResponse())); + + render(); + + await waitFor(() => { + expect(screen.getByText(/500 pushes in last hour/)).toBeInTheDocument(); + expect(screen.getByText(/8,000 in 24h/)).toBeInTheDocument(); + expect(screen.getByText(/50,000 in 7d/)).toBeInTheDocument(); + }); + }); +}); diff --git a/client/src/components/pages/Teams/OtlpStats.tsx b/client/src/components/pages/Teams/OtlpStats.tsx new file mode 100644 index 0000000..f439b56 --- /dev/null +++ b/client/src/components/pages/Teams/OtlpStats.tsx @@ -0,0 +1,430 @@ +import { useState, useEffect, useCallback } from 'react'; +import { Activity, Lock, Pencil, ChevronDown, ChevronUp } from 'lucide-react'; +import { getTeamOtlpStats, updateApiKeyRateLimit } from '../../../api/otlpStats'; +import type { OtlpStatsResponse, OtlpApiKeyStats } from '../../../types/otlpStats'; +import { ApiKeyUsageChart } from '../../Charts'; +import Modal from '../../common/Modal'; +import { formatRelativeTime } from '../../../utils/formatting'; +import teamStyles from './Teams.module.css'; +import styles from './OtlpStats.module.css'; + +const DEFAULT_RATE_LIMIT_RPM = 150_000; + +interface OtlpStatsProps { + teamId: string; + canManage?: boolean; +} + +function OtlpStats({ teamId, canManage = false }: OtlpStatsProps) { + const [data, setData] = useState(null); + const [isLoading, setIsLoading] = useState(true); + const [error, setError] = useState(null); + const [expandedCharts, setExpandedCharts] = useState>(new Set()); + + // Rate limit edit dialog state + const [editRateLimitKey, setEditRateLimitKey] = useState(null); + const [rateLimitInput, setRateLimitInput] = useState(''); + const [rateLimitError, setRateLimitError] = useState(null); + const [isSavingRateLimit, setIsSavingRateLimit] = useState(false); + + const loadStats = useCallback(async () => { + try { + setIsLoading(true); + const result = await getTeamOtlpStats(teamId); + setData(result); + setError(null); + } catch (err) { + setError(err instanceof Error ? err.message : 'Failed to load OTLP stats'); + } finally { + setIsLoading(false); + } + }, [teamId]); + + useEffect(() => { + loadStats(); + }, [loadStats]); + + const toggleChart = useCallback((keyId: string) => { + setExpandedCharts(prev => { + const next = new Set(prev); + if (next.has(keyId)) next.delete(keyId); + else next.add(keyId); + return next; + }); + }, []); + + const openRateLimitDialog = (key: OtlpApiKeyStats) => { + setEditRateLimitKey(key); + setRateLimitInput(key.rate_limit_is_custom ? String(key.rate_limit_rpm) : ''); + setRateLimitError(null); + }; + + const closeRateLimitDialog = () => { + setEditRateLimitKey(null); + setRateLimitInput(''); + setRateLimitError(null); + }; + + const validateRateLimitInput = (value: string): string | null => { + if (value === '') return null; + const num = Number(value); + if (!Number.isInteger(num) || num <= 0) return 'Must be a positive integer'; + if (num > 1_500_000) return 'Cannot exceed 1,500,000 req/min'; + return null; + }; + + const handleSaveRateLimit = async () => { + if (!editRateLimitKey) return; + const trimmed = rateLimitInput.trim(); + const validationError = validateRateLimitInput(trimmed); + if (validationError) { + setRateLimitError(validationError); + return; + } + const newLimit = trimmed === '' ? null : Number(trimmed); + try { + setIsSavingRateLimit(true); + await updateApiKeyRateLimit(teamId, editRateLimitKey.id, newLimit); + closeRateLimitDialog(); + await loadStats(); + } catch (err) { + setRateLimitError(err instanceof Error ? err.message : 'Failed to update rate limit'); + } finally { + setIsSavingRateLimit(false); + } + }; + + const handleResetToDefault = async () => { + if (!editRateLimitKey) return; + try { + setIsSavingRateLimit(true); + await updateApiKeyRateLimit(teamId, editRateLimitKey.id, null); + closeRateLimitDialog(); + await loadStats(); + } catch (err) { + setRateLimitError(err instanceof Error ? err.message : 'Failed to reset rate limit'); + } finally { + setIsSavingRateLimit(false); + } + }; + + if (isLoading) { + return ( +
+
+
+ Loading OTLP stats... +
+
+ ); + } + + if (error) { + return ( +
+
+

{error}

+ +
+
+ ); + } + + if (!data) return null; + + const { services, apiKeys, summary } = data; + + return ( +
+
+
+

OTLP Push Stats

+

+ Status of services receiving data via OTLP push. +

+
+
+ + {/* Summary Cards */} +
+
+
{summary.total_otlp_services}
+
Total OTLP Services
+
+
+
{summary.active_services}
+
Active
+
+
0 ? styles.summaryCardError : styles.summaryCard}> +
{summary.services_with_errors}
+
With Errors
+
+
0 ? styles.summaryCardWarning : styles.summaryCard}> +
{summary.services_never_pushed}
+
Never Pushed
+
+
+ + {/* Services Table */} + {services.length === 0 ? ( +
+ +

No OTLP services configured for this team.

+
+ ) : ( + <> +

Services

+
+ + + + + + + + + + + + + {services.map(s => ( + + + + + + + + + ))} + +
NameStatusLast PushErrors (24h)DependenciesWarnings
+ {s.name} + {!s.is_active && ( + + Inactive + + )} + + {s.last_push_success === null ? ( + Never pushed + ) : s.last_push_success ? ( + OK + ) : ( + Error + )} + + {s.last_push_at ? formatRelativeTime(s.last_push_at) : '—'} + + {s.errors_24h > 0 ? ( + {s.errors_24h} + ) : ( + '0' + )} + {s.dependency_count} + {s.last_push_warnings && s.last_push_warnings.length > 0 ? ( +
    + {s.last_push_warnings.map((w, i) => ( +
  • {w}
    • + ))} +
+ ) : ( + '—' + )} +
+
+ + )} + + {/* API Keys Section */} + {apiKeys.length > 0 && ( + <> +

API Keys

+
+ {apiKeys.map(k => ( +
+
+
+ + {k.name} + {k.rejected_24h > 0 && ( + Approaching limit + )} + {k.rejected_24h === 0 && k.rejected_7d > 0 && ( + + {k.rejected_7d.toLocaleString()} rejected in 7d + + )} + + {k.key_prefix}... +
+ + {k.last_used_at ? `Last used ${formatRelativeTime(k.last_used_at)}` : 'Never used'} + +
+ + {/* Usage summary row */} +
+ + {k.usage_1h.toLocaleString()} pushes in last hour + {' \u00b7 '}{k.usage_24h.toLocaleString()} in 24h + {' \u00b7 '}{k.usage_7d.toLocaleString()} in 7d + + {k.rejected_24h > 0 && ( + + {k.rejected_24h.toLocaleString()} rejected in 24h + + )} +
+ + {/* Rate limit display */} +
+ + Rate limit: {k.rate_limit_rpm === 0 + ? 'Unlimited' + : `${k.rate_limit_rpm.toLocaleString()} req/min`} + {' '} + + {k.rate_limit_rpm === 0 + ? '(admin)' + : k.rate_limit_is_custom ? '(custom)' : '(default)'} + + + {k.rate_limit_admin_locked && ( + + + + )} + {canManage && !k.rate_limit_admin_locked && k.rate_limit_rpm !== 0 && ( + + )} + +
+ + {/* Expandable usage chart */} + {expandedCharts.has(k.id) && ( +
+ +
+ )} +
+ ))} +
+ + )} + + {/* Error details */} + {services.some(s => s.last_push_error) && ( + <> +

Recent Errors

+
+ + + + + + + + + {services + .filter(s => s.last_push_error) + .map(s => ( + + + + + ))} + +
ServiceError
{s.name} + {s.last_push_error} +
+
+ + )} + + {editRateLimitKey && ( +
+

+ Set the rate limit for {editRateLimitKey.name} ({editRateLimitKey.key_prefix}...). + Leave empty to use the system default ({DEFAULT_RATE_LIMIT_RPM.toLocaleString()} req/min). +

+ + { + setRateLimitInput(e.target.value); + setRateLimitError(null); + }} + placeholder={String(DEFAULT_RATE_LIMIT_RPM)} + className={styles.rateLimitDialogInput} + min={1} + disabled={isSavingRateLimit} + onKeyDown={(e) => { + if (e.key === 'Enter') handleSaveRateLimit(); + }} + /> + {rateLimitError && ( +

{rateLimitError}

+ )} +
+ +
+ + +
+
+
+ )} + +
+ ); +} + +export default OtlpStats; diff --git a/client/src/components/pages/Teams/TeamDetail.tsx b/client/src/components/pages/Teams/TeamDetail.tsx index bf0c8d2..ca21d4d 100644 --- a/client/src/components/pages/Teams/TeamDetail.tsx +++ b/client/src/components/pages/Teams/TeamDetail.tsx @@ -14,6 +14,8 @@ import AlertChannels from './AlertChannels'; import AlertRules from './AlertRules'; import AlertHistory from './AlertHistory'; import AlertMutes from './AlertMutes'; +import ApiKeys from './ApiKeys'; +import OtlpStats from './OtlpStats'; import TeamOverviewStats from './TeamOverviewStats'; import ManifestList from '../Manifest/ManifestList'; import { useAlertChannels } from '../../../hooks/useAlertChannels'; @@ -149,6 +151,8 @@ function TeamDetail() { Services ({team.services.length}) Alerts Config + {canManageAlerts && API Keys} + {canManageAlerts && OTLP} {/* Overview Tab */} @@ -406,6 +410,19 @@ function TeamDetail() {
+ + {/* API Keys Tab */} + {canManageAlerts && ( + + + + )} + + {canManageAlerts && ( + + + + )} { expect(screen.getByTestId('user')).toHaveTextContent('New User'); }); }); + + it('clears user when auth:expired event is dispatched', async () => { + mockFetch.mockResolvedValueOnce({ + ok: true, + json: () => Promise.resolve({ id: '1', name: 'Test User', role: 'user' }), + }); + + render( + + + + ); + + await waitFor(() => { + expect(screen.getByTestId('authenticated')).toHaveTextContent('yes'); + }); + + act(() => { + window.dispatchEvent(new Event('auth:expired')); + }); + + expect(screen.getByTestId('authenticated')).toHaveTextContent('no'); + expect(screen.getByTestId('user')).toHaveTextContent('none'); + }); }); describe('useAuth', () => { diff --git a/client/src/contexts/AuthContext.tsx b/client/src/contexts/AuthContext.tsx index 4c64c76..f57ac4d 100644 --- a/client/src/contexts/AuthContext.tsx +++ b/client/src/contexts/AuthContext.tsx @@ -48,6 +48,12 @@ export function AuthProvider({ children }: AuthProviderProps) { checkAuth(); }, []); + useEffect(() => { + const handleExpired = () => setUser(null); + window.addEventListener('auth:expired', handleExpired); + return () => window.removeEventListener('auth:expired', handleExpired); + }, []); + const login = () => { // Redirect to OIDC login with return URL const returnTo = encodeURIComponent(window.location.pathname); diff --git a/client/src/hooks/useManageAssociations.test.ts b/client/src/hooks/useManageAssociations.test.ts index 4582bb0..f006eb4 100644 --- a/client/src/hooks/useManageAssociations.test.ts +++ b/client/src/hooks/useManageAssociations.test.ts @@ -24,6 +24,7 @@ function makeService(overrides = {}) { health_endpoint: 'http://localhost:3000/health', metrics_endpoint: null, schema_config: null, + health_endpoint_format: 'default' as const, is_active: 1, last_poll_success: 1, last_poll_error: null, @@ -105,6 +106,7 @@ function makeAssociation(overrides = {}) { health_endpoint: 'http://localhost:3001/health', metrics_endpoint: null, schema_config: null, + health_endpoint_format: 'default' as const, is_active: 1, last_poll_success: 1, last_poll_error: null, diff --git a/client/src/types/graph.ts b/client/src/types/graph.ts index 615d86d..ea694d5 100644 --- a/client/src/types/graph.ts +++ b/client/src/types/graph.ts @@ -54,6 +54,9 @@ export interface GraphEdgeData { impact?: string | null; effectiveContact?: string | null; skipped?: boolean; + discoverySource?: 'manual' | 'otlp_metric' | 'otlp_trace'; + isAutoSuggested?: boolean; + associationId?: string | null; [key: string]: unknown; } diff --git a/client/src/types/otlpStats.ts b/client/src/types/otlpStats.ts new file mode 100644 index 0000000..1bec232 --- /dev/null +++ b/client/src/types/otlpStats.ts @@ -0,0 +1,87 @@ +export interface OtlpServiceStats { + id: string; + name: string; + is_active: number; + last_push_success: number | null; + last_push_error: string | null; + last_push_warnings: string[] | null; + last_push_at: string | null; + dependency_count: number; + errors_24h: number; + schema_config: string | null; +} + +export interface OtlpApiKeyStats { + id: string; + name: string; + key_prefix: string; + last_used_at: string | null; + created_at: string; + rate_limit_rpm: number; + rate_limit_is_custom: boolean; + rate_limit_admin_locked: boolean; + usage_1h: number; + usage_24h: number; + usage_7d: number; + rejected_24h: number; + rejected_7d: number; +} + +export interface ApiKeyUsageBucket { + bucket_start: string; + push_count: number; + rejected_count: number; +} + +export interface ApiKeyUsageResponse { + api_key_id: string; + granularity: 'minute' | 'hour'; + from: string; + to: string; + buckets: ApiKeyUsageBucket[]; +} + +export interface OtlpStatsSummary { + total_otlp_services: number; + active_services: number; + services_with_errors: number; + services_never_pushed: number; +} + +export interface OtlpStatsResponse { + services: OtlpServiceStats[]; + apiKeys: OtlpApiKeyStats[]; + summary: OtlpStatsSummary; +} + +export interface AdminOtlpTeamStats { + team_id: string; + team_name: string; + services: OtlpServiceStats[]; + apiKeys: OtlpApiKeyStats[]; +} + +export interface AdminOtlpStatsSummary extends OtlpStatsSummary { + total_teams: number; +} + +export interface AdminOtlpStatsResponse { + teams: AdminOtlpTeamStats[]; + summary: AdminOtlpStatsSummary; +} + +export interface AdminOtlpUsageBucket { + api_key_id: string; + bucket_start: string; + granularity: 'minute' | 'hour'; + push_count: number; + rejected_count: number; + team_id: string; + key_name: string; +} + +export interface AdminOtlpUsageResponse { + from: string; + to: string; + buckets: AdminOtlpUsageBucket[]; +} diff --git a/client/src/types/service.ts b/client/src/types/service.ts index 82295ac..30ca2be 100644 --- a/client/src/types/service.ts +++ b/client/src/types/service.ts @@ -1,5 +1,6 @@ // Health state values: 0=OK, 1=WARNING, 2=CRITICAL export type HealthState = 0 | 1 | 2; +export type HealthEndpointFormat = 'default' | 'schema' | 'prometheus' | 'otlp'; export type HealthStatus = | 'healthy' | 'warning' @@ -70,6 +71,14 @@ export interface SchemaMapping { }; } +// Metric schema config for Prometheus and OTLP custom metric/label mappings +export interface MetricSchemaConfig { + metrics: Record; // user metric name → depsera field + labels: Record; // user label/attribute name → depsera field + latency_unit?: 'ms' | 's'; + healthy_value?: number; // value that means healthy for the 'healthy' target (default: 1) +} + export interface Service { id: string; name: string; @@ -82,6 +91,7 @@ export interface Service { description?: string | null; last_poll_success: number | null; last_poll_error: string | null; + health_endpoint_format: HealthEndpointFormat; poll_warnings: string | null; manifest_managed?: number; manifest_key?: string | null; @@ -126,6 +136,7 @@ export interface CreateServiceInput { health_endpoint: string; metrics_endpoint?: string; schema_config?: string | null; + health_endpoint_format?: HealthEndpointFormat; } export interface UpdateServiceInput { @@ -135,6 +146,7 @@ export interface UpdateServiceInput { metrics_endpoint?: string; schema_config?: string | null; is_active?: boolean; + health_endpoint_format?: HealthEndpointFormat; } // Test schema mapping response types diff --git a/docs/manifest-schema.md b/docs/manifest-schema.md index a32b07b..a2b0826 100644 --- a/docs/manifest-schema.md +++ b/docs/manifest-schema.md @@ -55,11 +55,27 @@ Each entry in the `services` array defines a monitored service. |-------|------|----------|-------------| | `key` | string | Yes | Unique identifier within the manifest. Lowercase alphanumeric, hyphens, and underscores. Must start with a letter or digit. Max 128 characters. Pattern: `^[a-z0-9][a-z0-9_-]*$` | | `name` | string | Yes | Human-readable display name. | -| `health_endpoint` | string | Yes | HTTP or HTTPS URL to poll for health status. | +| `health_endpoint` | string | Yes* | HTTP or HTTPS URL to poll for health status. *Not required when `health_endpoint_format` is `otlp` (push-only). | | `description` | string | No | Service description. | | `metrics_endpoint` | string | No | HTTP or HTTPS URL for metrics. | -| `poll_interval_ms` | integer | No | Polling interval in milliseconds. Must be between **5,000** (5s) and **3,600,000** (1hr). Defaults to the server's configured default if omitted. | -| `schema_config` | object | No | Custom schema mapping for non-standard health endpoints. See the [Health Endpoint Spec](health-endpoint-spec.md) for details. | +| `poll_interval_ms` | integer | No | Polling interval in milliseconds. Must be between **5,000** (5s) and **3,600,000** (1hr). Defaults to the server's configured default if omitted. Must be `0` for OTLP format. | +| `schema_config` | object | No | Custom schema mapping for non-standard health endpoints. Shape depends on `health_endpoint_format` — see below. | +| `health_endpoint_format` | string | No | Format of the health endpoint. One of: `default`, `schema`, `prometheus`, `otlp`. Defaults to `default` if omitted. | + +### Health Endpoint Format + +The `health_endpoint_format` field controls how the service's health endpoint is parsed and how `schema_config` is validated: + +| Format | Description | `schema_config` shape | +|--------|-------------|----------------------| +| `default` | Standard Depsera health endpoint format. | Not applicable (ignored). | +| `schema` | Custom JSON schema mapping. | `SchemaMapping` — requires `root` and `fields` with `name` and `healthy`. See the [Health Endpoint Spec](health-endpoint-spec.md). | +| `prometheus` | Prometheus metrics scrape endpoint. | `MetricSchemaConfig` — requires `metrics` and `labels` mappings. | +| `otlp` | OpenTelemetry push-based ingestion. No polling needed. | `MetricSchemaConfig` — requires `metrics` and `labels` mappings. | + +**OTLP notes:** +- `health_endpoint` may be empty or omitted (OTLP is push-only) +- `poll_interval_ms` must be `0` if specified ### Service Key Rules @@ -437,6 +453,54 @@ A service using Spring Boot Actuator's health endpoint with custom schema mappin } ``` +### OTLP Service (Push-Based) + +An OTLP service that receives telemetry data via push — no health endpoint polling: + +```json +{ + "version": 1, + "services": [ + { + "key": "telemetry-collector", + "name": "Telemetry Collector", + "health_endpoint_format": "otlp", + "description": "Receives OTLP push data from instrumented services", + "schema_config": { + "metrics": { "up": "healthy", "request_duration_seconds": "latency" }, + "labels": { "service_name": "name", "service_type": "type" }, + "latency_unit": "s" + } + } + ] +} +``` + +Note: `health_endpoint` is omitted (not required for OTLP) and `poll_interval_ms` defaults to `0`. + +### Prometheus Service with Custom Schema + +A service using a Prometheus metrics endpoint with custom metric/label mappings: + +```json +{ + "version": 1, + "services": [ + { + "key": "node-exporter", + "name": "Node Exporter", + "health_endpoint": "https://node-exporter.example.com/metrics", + "health_endpoint_format": "prometheus", + "poll_interval_ms": 30000, + "schema_config": { + "metrics": { "up": "healthy", "node_load1": "latency" }, + "labels": { "instance": "name", "job": "type" } + } + } + ] +} +``` + ### Multiple Teams Sharing Dependencies Two separate team manifests demonstrating shared canonical names and aliases: diff --git a/docs/spec/02-data-model.md b/docs/spec/02-data-model.md index 82b9d64..31eb93a 100644 --- a/docs/spec/02-data-model.md +++ b/docs/spec/02-data-model.md @@ -18,6 +18,7 @@ erDiagram users ||--o{ team_members : "has membership" teams ||--o{ team_members : "has members" teams ||--o{ services : "owns" + teams ||--o{ spans : "owns" services ||--o{ dependencies : "reports" dependencies ||--o{ dependency_latency_history : "records" dependencies ||--o{ dependency_error_history : "records" @@ -26,6 +27,8 @@ erDiagram dependency_aliases }o..o{ dependencies : "resolves name" dependency_canonical_overrides }o..o{ dependencies : "overrides by canonical_name" users ||--o{ dependency_canonical_overrides : "updated_by" + users ||--o{ external_node_enrichment : "updated_by" + users ||--o{ app_settings : "updated_by" services ||--o{ status_change_events : "records" services ||--o{ service_poll_history : "records" ``` @@ -81,6 +84,7 @@ erDiagram | health_endpoint | TEXT | NOT NULL | | | metrics_endpoint | TEXT | | NULL | | schema_config | TEXT | | NULL | +| health_endpoint_format | TEXT | NOT NULL | `'default'` | | poll_interval_ms | INTEGER | NOT NULL | 30000 | | is_active | INTEGER | NOT NULL | 1 | | last_poll_success | INTEGER | | NULL | @@ -91,6 +95,8 @@ erDiagram **Indexes:** `idx_services_team_id` on (team_id) +**`health_endpoint_format` values:** `'default'` (standard proactive-deps JSON array), `'schema'` (custom schema mapping), `'prometheus'` (Prometheus text exposition format), `'otlp'` (OpenTelemetry push — no polling). OTLP services have `health_endpoint = ''` and `poll_interval_ms = 0`. + **Constraints:** `poll_interval_ms` validated at API level: min 5000, max 3600000. Team delete is RESTRICT (cannot delete team with services). ### dependencies @@ -115,6 +121,10 @@ erDiagram | error | TEXT | | NULL | | error_message | TEXT | | NULL | | skipped | INTEGER | NOT NULL | 0 | +| discovery_source | TEXT | NOT NULL | `'manual'` | +| user_display_name | TEXT | | NULL | +| user_description | TEXT | | NULL | +| user_impact | TEXT | | NULL | | last_checked | TEXT | | NULL | | last_status_change | TEXT | | NULL | | created_at | TEXT | NOT NULL | `datetime('now')` | @@ -126,6 +136,10 @@ erDiagram **`type` enum:** `database`, `rest`, `soap`, `grpc`, `graphql`, `message_queue`, `cache`, `file_system`, `smtp`, `other` +**`discovery_source` values:** `'manual'` (user-created or polled), `'otlp_metric'` (auto-created from OTLP metric push), `'otlp_trace'` (auto-discovered from trace spans). On upsert conflict, manual dependencies are never downgraded — if `discovery_source` is already `'manual'`, subsequent trace pushes preserve it. + +**User enrichment columns:** `user_display_name`, `user_description`, `user_impact` are user-managed overrides separate from the auto-detected `name`/`description`/`impact` columns. Trace pushes update auto-detected fields but never overwrite user enrichment. + **`health_state` values:** 0 = OK, 1 = WARNING, 2 = CRITICAL ### dependency_associations @@ -137,14 +151,18 @@ erDiagram | linked_service_id | TEXT | NOT NULL, FK → services.id CASCADE | | | linked_service_key | TEXT | | NULL | | association_type | TEXT | NOT NULL, CHECK (see below) | `'other'` | +| is_auto_suggested | INTEGER | NOT NULL | 0 | +| is_dismissed | INTEGER | NOT NULL | 0 | | created_at | TEXT | NOT NULL | `datetime('now')` | **Unique constraint:** `(dependency_id, linked_service_id)` -**Indexes:** `idx_dependency_associations_dependency_id`, `idx_dependency_associations_linked_service_id` +**Indexes:** `idx_dependency_associations_dependency_id`, `idx_dependency_associations_linked_service_id`, `idx_dep_assoc_auto_suggested` on (is_auto_suggested, is_dismissed) **`association_type` enum:** `api_call`, `database`, `message_queue`, `cache`, `other` +**Auto-suggestion columns:** `is_auto_suggested` is 1 when the association was automatically created from trace data (via `AutoAssociator`). Users can confirm (sets `is_auto_suggested=0`) or dismiss (sets `is_dismissed=1`). Dismissed associations are not re-suggested by subsequent trace pushes. Old dismissed associations are cleaned up by `DataRetentionService`. + ### dependency_latency_history | Column | Type | Constraints | Default | @@ -152,10 +170,21 @@ erDiagram | id | TEXT | PRIMARY KEY | | | dependency_id | TEXT | NOT NULL, FK → dependencies.id CASCADE | | | latency_ms | INTEGER | NOT NULL | | +| p50_ms | REAL | | NULL | +| p95_ms | REAL | | NULL | +| p99_ms | REAL | | NULL | +| min_ms | REAL | | NULL | +| max_ms | REAL | | NULL | +| request_count | INTEGER | | NULL | +| source | TEXT | NOT NULL | `'poll'` | | recorded_at | TEXT | NOT NULL | `datetime('now')` | **Indexes:** `idx_latency_history_dependency`, `idx_latency_history_time` +**`source` values:** `'poll'` (from health endpoint polling), `'otlp_gauge'` (from OTLP gauge metrics), `'otlp_histogram'` (from OTLP histogram metrics), `'otlp_trace'` (from trace span durations). Existing rows default to `'poll'`. + +**Percentile columns:** Nullable `REAL` columns populated when histogram data is available. `p50_ms`, `p95_ms`, `p99_ms` are computed via linear interpolation from histogram bucket boundaries. `min_ms` and `max_ms` are passthrough from histogram data points when available. `request_count` stores the total count from the histogram or sum data point. + ### dependency_error_history | Column | Type | Constraints | Default | @@ -236,6 +265,99 @@ Records dependency health status transitions detected during polling. `previous_ Records service-level poll success/failure transitions with deduplication. Only state changes are recorded (success→failure, failure→success, or error message change). A null `error` entry represents recovery (poll succeeded after prior failure). Displayed on the service detail page in the "Poll Issues" section. Subject to data retention cleanup. +### team_api_keys **[Implemented]** + +| Column | Type | Constraints | Default | +|---|---|---|---| +| id | TEXT | PRIMARY KEY | | +| team_id | TEXT | NOT NULL, FK → teams.id CASCADE | | +| name | TEXT | NOT NULL | | +| key_hash | TEXT | NOT NULL | | +| key_prefix | TEXT | NOT NULL | | +| last_used_at | TEXT | | NULL | +| created_at | TEXT | NOT NULL | `datetime('now')` | +| created_by | TEXT | FK → users.id | NULL | +| rate_limit_rpm | INTEGER | | NULL | +| rate_limit_admin_locked | INTEGER | NOT NULL | 0 | + +**Indexes:** `idx_team_api_keys_key_hash` UNIQUE on (key_hash), `idx_team_api_keys_team_id` on (team_id) + +Team-scoped API keys for authenticating OTLP push requests. `key_hash` stores SHA-256 of the raw API key (format: `dps_` + 32 random hex chars). `key_prefix` stores the first 8 characters for UI display (e.g., `dps_a1b2...`). The raw key is only returned once at creation time. Used by the `requireApiKeyAuth` middleware to authenticate `POST /v1/metrics` requests. `rate_limit_rpm`: NULL = system default (env `OTLP_PER_KEY_RATE_LIMIT_RPM`, default 150,000), 0 = unlimited (admin-only), N = custom rpm. `rate_limit_admin_locked`: 0 = unlocked (team can self-serve), 1 = admin has locked against team edits. + +### api_key_usage_buckets **[Implemented]** + +| Column | Type | Constraints | Default | +|---|---|---|---| +| api_key_id | TEXT | NOT NULL, PK | | +| bucket_start | TEXT | NOT NULL, PK | | +| granularity | TEXT | NOT NULL, PK, CHECK (`minute`, `hour`) | | +| push_count | INTEGER | NOT NULL | 0 | +| rejected_count | INTEGER | NOT NULL | 0 | + +**Primary Key:** (api_key_id, bucket_start, granularity) +**Indexes:** `idx_usage_buckets_key_start` on (api_key_id, bucket_start), `idx_usage_buckets_start` on (bucket_start) + +Time-series bucketed usage counters for API key push requests. No FK cascade by design — when a key is hard-deleted, orphaned usage rows are retained for 7 days then pruned by the retention job. Minute-granularity rows are retained 24 hours; hour-granularity rows are retained 30 days. `bucket_start` is an ISO 8601 UTC timestamp truncated to minute or hour (e.g., `2025-01-15T14:32:00` or `2025-01-15T14:00:00`). + +### spans **[Implemented]** + +| Column | Type | Constraints | Default | +|---|---|---|---| +| id | TEXT | PRIMARY KEY | | +| trace_id | TEXT | NOT NULL | | +| span_id | TEXT | NOT NULL | | +| parent_span_id | TEXT | | NULL | +| service_name | TEXT | NOT NULL | | +| team_id | TEXT | NOT NULL, FK → teams.id CASCADE | | +| name | TEXT | NOT NULL | | +| kind | INTEGER | NOT NULL | 0 | +| start_time | TEXT | NOT NULL | | +| end_time | TEXT | NOT NULL | | +| duration_ms | REAL | NOT NULL | | +| status_code | INTEGER | | 0 | +| status_message | TEXT | | NULL | +| attributes | TEXT | | NULL | +| resource_attributes | TEXT | | NULL | +| created_at | TEXT | NOT NULL | `datetime('now')` | + +**Indexes:** `idx_spans_trace_id` on (trace_id), `idx_spans_service_team` on (service_name, team_id), `idx_spans_start_time` on (start_time), `idx_spans_kind` on (kind), `idx_spans_created_at` on (created_at) + +Full span storage for OTLP trace data. Denormalized flat table — spans are write-heavy, read-occasionally. `service_name` is denormalized from resource attributes for fast per-service queries. `duration_ms` is precomputed from nanosecond timestamps. `attributes` and `resource_attributes` are JSON strings. ALL span types (CLIENT, SERVER, PRODUCER, CONSUMER, INTERNAL) are persisted; only CLIENT and PRODUCER feed into dependency discovery. + +**`kind` enum:** 0 = UNSPECIFIED, 1 = INTERNAL, 2 = SERVER, 3 = CLIENT, 4 = PRODUCER, 5 = CONSUMER (per OpenTelemetry spec). + +**`status_code` values:** 0 = UNSET, 1 = OK, 2 = ERROR. + +**Retention:** Configurable via `app_settings.span_retention_days` (default 7 days). Old spans are cleaned up by `DataRetentionService`. + +### app_settings **[Implemented]** + +| Column | Type | Constraints | Default | +|---|---|---|---| +| key | TEXT | PRIMARY KEY | | +| value | TEXT | NOT NULL | | +| updated_at | TEXT | | `datetime('now')` | +| updated_by | TEXT | FK → users.id | NULL | + +Admin-configurable application settings. Seeded with `span_retention_days = '7'` on migration. Used by `DataRetentionService` for configurable span cleanup window and exposed via `GET/PUT /api/admin/settings/span-retention`. + +### external_node_enrichment **[Implemented]** + +| Column | Type | Constraints | Default | +|---|---|---|---| +| id | TEXT | PRIMARY KEY | | +| canonical_name | TEXT | NOT NULL, UNIQUE | | +| display_name | TEXT | | NULL | +| description | TEXT | | NULL | +| impact | TEXT | | NULL | +| contact | TEXT | | NULL | +| service_type | TEXT | | NULL | +| created_at | TEXT | NOT NULL | `datetime('now')` | +| updated_at | TEXT | NOT NULL | `datetime('now')` | +| updated_by | TEXT | FK → users.id | NULL | + +Org-wide enrichment for virtual external nodes in the dependency graph. Keyed by `canonical_name` to match `ExternalNodeBuilder`'s grouping logic (lowercase + trim). Not team-scoped — matches the cross-team external node deduplication via `SHA-256(normalized_name)`. `contact` is a JSON string (arbitrary contact object). `service_type` overrides the inferred type from dependency edges. Applied to graph external nodes by `GraphService` during graph building. + ## Type Enumerations ```typescript @@ -246,6 +368,9 @@ type AggregatedHealthStatus = 'healthy' | 'warning' | 'critical' | 'unknown'; type DependencyType = 'database' | 'rest' | 'soap' | 'grpc' | 'graphql' | 'message_queue' | 'cache' | 'file_system' | 'smtp' | 'other'; type AssociationType = 'api_call' | 'database' | 'message_queue' | 'cache' | 'other'; +type HealthEndpointFormat = 'default' | 'schema' | 'prometheus' | 'otlp'; +type DiscoverySource = 'manual' | 'otlp_metric' | 'otlp_trace'; +type LatencySource = 'poll' | 'otlp_gauge' | 'otlp_histogram' | 'otlp_trace'; type AlertSeverityFilter = 'critical' | 'warning' | 'all'; type DriftType = 'field_change' | 'service_removal'; type DriftFlagStatus = 'pending' | 'dismissed' | 'accepted' | 'resolved'; @@ -284,13 +409,41 @@ Key-value store for runtime-configurable admin settings. Records admin actions (role changes, user deactivation/reactivation, team CRUD, team member changes, service CRUD, canonical override management, per-instance override management). -**Audit actions:** `user.created`, `user.role_changed`, `user.deactivated`, `user.reactivated`, `user.password_reset`, `team.created`, `team.updated`, `team.deleted`, `team.member_added`, `team.member_removed`, `team.member_role_changed`, `service.created`, `service.updated`, `service.deleted`, `external_service.created`, `external_service.updated`, `external_service.deleted`, `settings.updated`, `canonical_override.upserted`, `canonical_override.deleted`, `dependency_override.updated`, `dependency_override.cleared`, `alert_mute.created`, `alert_mute.deleted` +**Audit actions:** `user.created`, `user.role_changed`, `user.deactivated`, `user.reactivated`, `user.password_reset`, `team.created`, `team.updated`, `team.deleted`, `team.member_added`, `team.member_removed`, `team.member_role_changed`, `service.created`, `service.updated`, `service.deleted`, `external_service.created`, `external_service.updated`, `external_service.deleted`, `settings.updated`, `canonical_override.upserted`, `canonical_override.deleted`, `dependency_override.updated`, `dependency_override.cleared`, `alert_mute.created`, `alert_mute.deleted`, `api_key.created`, `api_key.revoked` -**Resource types:** `user`, `team`, `service`, `external_service`, `settings`, `canonical_override`, `dependency`, `alert_mute` +**Resource types:** `user`, `team`, `service`, `external_service`, `settings`, `canonical_override`, `dependency`, `alert_mute`, `team_api_key` ### schema_config (on services) **[Implemented]** -Custom health endpoint schema configuration stored as a nullable `schema_config TEXT` column on the `services` table (JSON string of `SchemaMapping`). Services without a mapping default to proactive-deps format. +Custom health endpoint schema configuration stored as a nullable `schema_config TEXT` column on the `services` table. The column stores a JSON string whose shape depends on the service's `health_endpoint_format`: + +- **`schema` format:** `SchemaMapping` — root path and field mappings for custom JSON health endpoints. +- **`prometheus` / `otlp` formats:** `MetricSchemaConfig` — custom metric name and label/attribute name mappings (see below). +- **`default` format:** Always `null`. + +Services without a mapping default to proactive-deps format. + +### MetricSchemaConfig **[Implemented]** + +Configuration type for customizing Prometheus and OTLP metric/label mappings. Stored in the `schema_config` TEXT column on `services` when `health_endpoint_format` is `'prometheus'` or `'otlp'`. + +```typescript +interface MetricSchemaConfig { + metrics: Record; // user metric name → depsera field + labels: Record; // user label/attribute name → depsera field + latency_unit?: 'ms' | 's'; // default 'ms' +} +``` + +**Valid metric targets:** `state`, `healthy`, `latency`, `code`, `skipped` + +**Valid label targets:** `name`, `type`, `impact`, `description`, `errorMessage` + +**Merge behavior:** User-provided mappings in `metrics` and `labels` override the defaults. When a user maps a custom name to a target field (e.g., `{ "my_latency": "latency" }`), the default entry for that target is removed and replaced. Entries not overridden retain their defaults. + +**Validation:** `validateMetricSchemaConfig()` in `server/src/utils/validation.ts`. Both `metrics` and `labels` are required objects. Each value must be a valid target string. Duplicate targets within `metrics` or `labels` are rejected. `latency_unit` must be `'ms'` or `'s'` if provided. Schema config validation is format-aware: `prometheus`/`otlp` formats validate as `MetricSchemaConfig`, `schema` format validates as `SchemaMapping`, `default` format sets `schema_config` to `null`. + +**Type guard:** `isMetricSchemaConfig()` in `server/src/services/polling/metricSchemaUtils.ts` distinguishes `MetricSchemaConfig` (has `metrics`/`labels`) from `SchemaMapping` (has `root`/`fields`). ### alert_channels **[Implemented]** @@ -504,10 +657,48 @@ Contains all types specific to the manifest sync engine: ### Updated existing interfaces -- `Service`: added `manifest_key: string | null`, `manifest_managed: number`, `manifest_last_synced_values: string | null` +- `Service`: added `manifest_key: string | null`, `manifest_managed: number`, `manifest_last_synced_values: string | null`, `health_endpoint_format: HealthEndpointFormat` - `DependencyAlias`: added `manifest_team_id: string | null` - `DependencyCanonicalOverride`: added `team_id: string | null`, `manifest_managed: number` -- `DependencyAssociation`: added `manifest_managed: number` +- `DependencyAssociation`: added `manifest_managed: number`, `is_auto_suggested: number`, `is_dismissed: number` +- `Dependency`: added `discovery_source: DiscoverySource`, `user_display_name: string | null`, `user_description: string | null`, `user_impact: string | null` + +### Trace discovery types **[Implemented]** + +#### `server/src/db/types.ts` — Span and enrichment types + +- `DiscoverySource`: `'manual' | 'otlp_metric' | 'otlp_trace'` +- `Span`: DB row type for `spans` table — `id`, `trace_id`, `span_id`, `parent_span_id`, `service_name`, `team_id`, `name`, `kind`, `start_time`, `end_time`, `duration_ms`, `status_code`, `status_message`, `attributes`, `resource_attributes`, `created_at` +- `CreateSpanInput`: input type for `SpanStore.bulkInsert()` — same fields as `Span` minus `id` and `created_at`, with optional `kind`, `status_code`, `status_message`, `attributes`, `resource_attributes` +- `ExternalNodeEnrichment`: DB row type for `external_node_enrichment` table +- `UpsertExternalNodeEnrichmentInput`: input type for `ExternalNodeEnrichmentStore.upsert()` — requires `canonical_name`, all other fields optional + +#### Extended `ProactiveDepsStatus.health` + +Added optional `percentiles` field: + +```typescript +percentiles?: { + p50?: number; + p95?: number; + p99?: number; + min?: number; + max?: number; + requestCount?: number; +} +``` + +Populated when histogram data is available from OTLP metric pushes. Used by `DependencyUpsertService` to call `recordWithPercentiles()` instead of `record()`. + +#### `LatencyBucket` extension + +Extended with optional percentile averages for time-bucketed latency queries: + +```typescript +avg_p50?: number | null; +avg_p95?: number | null; +avg_p99?: number | null; +``` ### ManifestValidator **[Implemented]** @@ -621,5 +812,13 @@ Contains all types specific to the manifest sync engine: | 030 | add_alert_delay | Adds `alert_delay_minutes INTEGER` to `alert_rules` for requiring continuous unhealthy state before alerting | | 031 | add_alert_mutes | Creates `alert_mutes` table with CHECK constraint, unique indexes; rebuilds `alert_history` to add 'muted' to status CHECK | | 032 | add_service_mutes | Adds `service_id` column to `alert_mutes`; rebuilds table with updated CHECK constraint (exactly one of three targets); adds `idx_alert_mutes_service` unique index | +| 034 | add_otel_sources | Adds `health_endpoint_format TEXT NOT NULL DEFAULT 'default'` to `services`; backfills `'schema'` for services with `schema_config`; creates `team_api_keys` table with unique index on `key_hash` and index on `team_id` | +| 035 | api_key_rate_limit_columns | Adds `rate_limit_rpm INTEGER` (NULL = system default, 0 = unlimited, N = custom rpm) and `rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0` to `team_api_keys` | +| 036 | api_key_usage_buckets | Creates `api_key_usage_buckets` table (composite PK: api_key_id, bucket_start, granularity) with indexes `idx_usage_buckets_key_start` and `idx_usage_buckets_start`; no FK cascade by design — orphaned rows pruned by retention | +| 037 | add_trace_discovery | Adds `discovery_source TEXT NOT NULL DEFAULT 'manual'` to `dependencies` with backfill (`'otlp_metric'` for OTLP services); adds `user_display_name`, `user_description`, `user_impact` to `dependencies`; adds `is_auto_suggested INTEGER NOT NULL DEFAULT 0`, `is_dismissed INTEGER NOT NULL DEFAULT 0` to `dependency_associations`; creates index `idx_dep_assoc_auto_suggested` | +| 038 | add_external_node_enrichment | Creates `external_node_enrichment` table with UNIQUE on `canonical_name` and FK `updated_by → users.id` | +| 039 | add_percentile_latency | Adds `p50_ms REAL`, `p95_ms REAL`, `p99_ms REAL`, `min_ms REAL`, `max_ms REAL`, `request_count INTEGER`, `source TEXT NOT NULL DEFAULT 'poll'` to `dependency_latency_history` | +| 040 | add_span_storage | Creates `spans` table with FK `team_id → teams.id CASCADE`; indexes on `trace_id`, `(service_name, team_id)`, `start_time`, `kind`, `created_at` | +| 041 | add_span_retention_setting | Creates `app_settings` table with FK `updated_by → users.id`; seeds `span_retention_days = '7'` | Migrations are tracked in a `_migrations` table (`id TEXT PK`, `name TEXT`, `applied_at TEXT`). Each migration runs in a transaction. diff --git a/docs/spec/03-auth.md b/docs/spec/03-auth.md index d9e169f..517ec33 100644 --- a/docs/spec/03-auth.md +++ b/docs/spec/03-auth.md @@ -83,9 +83,14 @@ interface SessionData { | Expired session cleanup | Every 15 minutes | | resave | `false` | | saveUninitialized | `false` | +| rolling | `true` (resets maxAge on each response — session expires after 24h of inactivity, not 24h from login) | **Startup warning:** `warnInsecureCookies()` logs a warning if `NODE_ENV` is not `development` and neither `REQUIRE_HTTPS` nor `TRUST_PROXY` is configured, since the `'auto'` secure flag will resolve to `false`, sending cookies over HTTP. +### Session Expiration Handling (Client) + +When an API call returns HTTP 401, `handleResponse()` dispatches a `window` event (`auth:expired`). The `AuthProvider` listens for this event and clears the user state, which causes `ProtectedRoute` to redirect to `/login`. This ensures users are automatically redirected to the login page when their session expires mid-use, without requiring a page reload. + ## 3.4 Session Secret Validation **[Implemented]** On startup, `validateSessionSecret()` checks `SESSION_SECRET`: @@ -167,3 +172,30 @@ A local authentication mode for zero-external-dependency deployment: - `GET /api/auth/mode` returns `{ mode: "oidc" | "local" }` (public, no auth required) - Client login page conditionally renders local auth form or OIDC button based on `GET /api/auth/mode` **[Implemented]** (PRO-100) - Admin can create users and reset passwords via API **[Implemented]** (PRO-101). `POST /api/users` creates a local user; `PUT /api/users/:id/password` resets password. Both gated by `requireLocalAuth`. + +## 3.8 API Key Authentication **[Implemented]** + +Team-scoped API keys for authenticating OTLP metric push requests from OpenTelemetry collectors. + +### Key Format + +- Format: `dps_` + 32 random hex characters (e.g., `dps_a1b2c3d4e5f6...`) +- Generated via `crypto.randomBytes(16).toString('hex')` +- Stored as SHA-256 hash (`key_hash`); raw key returned only once at creation +- `key_prefix` stores first 8 characters for UI display + +### `requireApiKeyAuth` Middleware + +Authenticates requests via `Authorization: Bearer dps_...` header: + +1. Extracts token from `Authorization: Bearer ` header +2. Computes SHA-256 hash of the raw token +3. Looks up `team_api_keys` by `key_hash` +4. On success: sets `req.apiKeyTeamId` to the key's `team_id` and `req.apiKeyId` to the key's `id`, updates `last_used_at` asynchronously +5. On failure: returns `401 { error: "..." }` + +**Key differences from session auth:** +- Bypasses CSRF validation (collectors don't have CSRF tokens) +- Bypasses session middleware (mounted before session layer in middleware order) +- Does not set `req.user` — only `req.apiKeyTeamId` and `req.apiKeyId` +- Used exclusively for `POST /v1/metrics` (OTLP receiver endpoint) diff --git a/docs/spec/04-api-reference.md b/docs/spec/04-api-reference.md index 721ab37..c45d93a 100644 --- a/docs/spec/04-api-reference.md +++ b/docs/spec/04-api-reference.md @@ -63,10 +63,16 @@ Rate limited: 20 requests/minute per IP. ```json { "url": "https://example.com/health (required, SSRF-validated)", - "schema_config": "SchemaMapping object or JSON string (required)" + "schema_config": "SchemaMapping or MetricSchemaConfig object, or JSON string (required for 'schema' format, optional for 'prometheus')", + "format": "'default' | 'schema' | 'prometheus' | 'otlp' (optional, default 'schema')" } ``` +**Format-specific behavior:** +- `'schema'`/`'default'`: fetches JSON, parses with SchemaMapper (existing behavior). `schema_config` is a `SchemaMapping`. +- `'prometheus'`: fetches with `Accept: text/plain; version=0.0.4`, parses with PrometheusParser. Optional `schema_config` (`MetricSchemaConfig`) for custom metric/label name mappings. +- `'otlp'`: returns error — OTLP services receive pushed metrics and cannot be tested via URL + **POST /api/services/test-schema response:** ```json @@ -87,13 +93,20 @@ On parse failure: `{ success: false, dependencies: [], warnings: ["error message { "name": "string (required)", "team_id": "uuid (required)", - "health_endpoint": "url (required, SSRF-validated)", + "health_endpoint": "url (required for polled formats, SSRF-validated)", + "health_endpoint_format": "'default' | 'schema' | 'prometheus' | 'otlp' (optional, default 'default')", "metrics_endpoint": "url (optional)", - "schema_config": "SchemaMapping object or null (optional, see Section 12.5)", + "schema_config": "SchemaMapping or MetricSchemaConfig object, or null (optional, see format-specific rules)", "poll_interval_ms": "number (optional, default 30000, min 5000, max 3600000)" } ``` +**Format-specific rules:** +- `'default'`: health endpoint URL required, `schema_config` set to `null` +- `'schema'`: health endpoint URL required, `schema_config` required (`SchemaMapping`) +- `'prometheus'`: health endpoint URL required, fetched with `Accept: text/plain; version=0.0.4`. Optional `schema_config` (`MetricSchemaConfig`) for custom metric/label name mappings. +- `'otlp'`: health endpoint URL not required (push-only), `poll_interval_ms` set to 0, service receives metrics via `POST /v1/metrics`. Optional `schema_config` (`MetricSchemaConfig`) for custom metric/attribute name mappings. + **GET /api/services/:id response:** ```json @@ -103,6 +116,7 @@ On parse failure: `{ success: false, dependencies: [], warnings: ["error message "team_id": "uuid", "team": { "id": "uuid", "name": "Platform", "description": "..." }, "health_endpoint": "https://payment-svc/health", + "health_endpoint_format": "default", "metrics_endpoint": null, "schema_config": null, "poll_interval_ms": 30000, @@ -164,6 +178,163 @@ On parse failure: `{ success: false, dependencies: [], warnings: ["error message - Cannot delete team with services (409 Conflict) - Cannot add existing member (409 Conflict) +### Team API Keys **[Implemented]** + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/api/teams/:id/api-keys` | requireTeamLead | List API keys for team. Never returns raw key or hash. | +| POST | `/api/teams/:id/api-keys` | requireTeamLead | Create API key. Returns raw key once. | +| DELETE | `/api/teams/:id/api-keys/:keyId` | requireTeamLead | Revoke API key. Returns 204. | + +**POST /api/teams/:id/api-keys request:** + +```json +{ + "name": "string (required, descriptive name for the key)" +} +``` + +**POST /api/teams/:id/api-keys response:** + +```json +{ + "id": "uuid", + "team_id": "uuid", + "name": "Production Collector", + "key_prefix": "dps_a1b2", + "raw_key": "dps_a1b2c3d4e5f6... (shown ONCE, never retrievable again)", + "created_at": "2026-03-10T10:00:00.000Z", + "created_by": "user-uuid" +} +``` + +**GET /api/teams/:id/api-keys response:** + +```json +[ + { + "id": "uuid", + "team_id": "uuid", + "name": "Production Collector", + "key_prefix": "dps_a1b2", + "last_used_at": "2026-03-15T08:30:00.000Z", + "created_at": "2026-03-10T10:00:00.000Z", + "created_by": "user-uuid" + } +] +``` + +**Audit actions:** `api_key.created`, `api_key.revoked` (resource type: `team_api_key`). Audit detail includes `key_prefix` and `team_id`. + +### Team API Key Rate Limits **[Implemented]** + +| Method | Path | Auth | Description | +|---|---|---|---| +| PATCH | `/api/teams/:id/api-keys/:keyId/rate-limit` | requireTeamLead | Update per-key rate limit. | + +**PATCH /api/teams/:id/api-keys/:keyId/rate-limit request:** + +```json +{ + "rate_limit_rpm": "number | null (null = reset to system default)" +} +``` + +**Validation:** +- `rate_limit_rpm` must be a positive integer (1–1,500,000) or `null` +- Cannot set to `0` (unlimited) — only admins can do this (400) +- Returns 403 if the key's `rate_limit_admin_locked` is true +- Returns 404 if the key does not belong to the specified team + +**Response:** Returns the updated API key (excluding `key_hash`). Evicts the in-memory rate limit bucket, forcing re-initialization on the next request. + +### Team API Key Usage **[Implemented]** + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/api/teams/:id/api-keys/:keyId/usage` | requireTeamLead | Get usage time series for an API key. | + +**Query parameters:** +- `granularity`: `minute` or `hour` (default: `minute`) +- `from`: ISO 8601 timestamp (default: 24h ago for minute, 30d ago for hour) +- `to`: ISO 8601 timestamp (default: now) + +**GET /api/teams/:id/api-keys/:keyId/usage response:** + +```json +{ + "api_key_id": "uuid", + "granularity": "minute", + "from": "2026-04-04T10:00:00.000Z", + "to": "2026-04-05T10:00:00.000Z", + "buckets": [ + { + "api_key_id": "uuid", + "bucket_start": "2026-04-04T10:00:00", + "granularity": "minute", + "push_count": 42, + "rejected_count": 0 + } + ] +} +``` + +### Team OTLP Stats **[Implemented]** + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/api/teams/:id/otlp-stats` | requireAuth | OTLP dashboard stats for a team. | + +**GET /api/teams/:id/otlp-stats response:** + +```json +{ + "services": [ + { + "id": "uuid", + "name": "my-otel-service", + "is_active": 1, + "last_push_success": 1, + "last_push_error": null, + "last_push_warnings": null, + "last_push_at": "2026-04-05T08:30:00.000Z", + "dependency_count": 3, + "errors_24h": 0, + "schema_config": null + } + ], + "apiKeys": [ + { + "id": "uuid", + "name": "Production Collector", + "key_prefix": "dps_a1b2", + "last_used_at": "2026-04-05T08:30:00.000Z", + "created_at": "2026-03-10T10:00:00.000Z", + "rate_limit_rpm": 150000, + "rate_limit_is_custom": false, + "rate_limit_admin_locked": false, + "usage_1h": 120, + "usage_24h": 2880, + "usage_7d": 20160, + "rejected_24h": 0, + "rejected_7d": 0 + } + ], + "summary": { + "total_otlp_services": 2, + "active_services": 2, + "services_with_errors": 0, + "services_never_pushed": 0 + } +} +``` + +- `rate_limit_rpm`: effective rate limit (custom value if set, otherwise system default) +- `rate_limit_is_custom`: `true` if the key has a non-null `rate_limit_rpm` override +- `rate_limit_admin_locked`: `true` if admin has locked the key's rate limit +- Usage summaries (`usage_1h`, `usage_24h`, `usage_7d`) show total `push_count` in each window +- Rejection summaries (`rejected_24h`, `rejected_7d`) show total `rejected_count` (429s) in each window + ## 4.5 Users | Method | Path | Auth | Description | @@ -354,6 +525,10 @@ Transitions derived from `dependency_error_history`: error entries map to `"unhe | PUT | `/api/admin/settings` | requireAdmin | Update settings. Body: partial object of `{ key: value }` pairs. Validates values before persisting. | | GET | `/api/admin/manifests` | requireAdmin | List all teams with manifest config status, drift counts, and contact info. | | POST | `/api/admin/manifests/sync-all` | requireAdmin | Trigger sync for all enabled manifest configs. Returns per-team results. | +| PATCH | `/api/admin/api-keys/:keyId/rate-limit` | requireAdmin | Update any API key's rate limit and admin lock. | +| GET | `/api/admin/api-keys/:keyId/usage` | requireAdmin | Get usage time series for any API key. | +| GET | `/api/admin/otlp-usage` | requireAdmin | Cross-team aggregated OTLP usage overview. | +| GET | `/api/admin/otlp-stats` | requireAdmin | Workspace-wide OTLP stats grouped by team. | **GET /api/admin/audit-log response:** @@ -379,6 +554,94 @@ Transitions derived from `dependency_error_history`: error entries map to `"unhe } ``` +### Admin API Key Rate Limits **[Implemented]** + +**PATCH /api/admin/api-keys/:keyId/rate-limit request:** + +```json +{ + "rate_limit_rpm": "number | null (null = reset to system default, 0 = unlimited)", + "admin_locked": "boolean (optional, locks team from editing)" +} +``` + +**Validation:** +- `rate_limit_rpm` must be a non-negative integer or `null` +- Can set to `0` (unlimited) — admin-only privilege +- `admin_locked: true` prevents the owning team from changing the key's rate limit +- Both fields can be combined in a single request + +**Response:** Returns the updated API key (excluding `key_hash`). Evicts the in-memory rate limit bucket. + +### Admin API Key Usage **[Implemented]** + +**GET /api/admin/api-keys/:keyId/usage** — Same interface as team-level usage endpoint (section 4.4 Team API Key Usage) but without team membership check. Returns usage time series for any API key. + +### Admin OTLP Usage Overview **[Implemented]** + +**GET /api/admin/otlp-usage** — Cross-team aggregated OTLP usage. + +**Query parameters:** +- `from`: ISO 8601 timestamp (default: 7 days ago) +- `to`: ISO 8601 timestamp (default: now) + +**Response:** `{ from, to, buckets }` — hourly-granularity buckets across all keys and teams. + +### Admin OTLP Stats **[Implemented]** + +**GET /api/admin/otlp-stats** — Workspace-wide OTLP overview grouped by team. + +**GET /api/admin/otlp-stats response:** + +```json +{ + "teams": [ + { + "team_id": "uuid", + "team_name": "Platform", + "services": [ + { + "id": "uuid", + "name": "my-otel-service", + "is_active": 1, + "last_push_success": 1, + "last_push_error": null, + "last_push_warnings": null, + "last_push_at": "2026-04-05T08:30:00.000Z", + "dependency_count": 3, + "errors_24h": 0, + "schema_config": null + } + ], + "apiKeys": [ + { + "id": "uuid", + "name": "Production Collector", + "key_prefix": "dps_a1b2", + "last_used_at": "2026-04-05T08:30:00.000Z", + "created_at": "2026-03-10T10:00:00.000Z", + "rate_limit_rpm": 150000, + "rate_limit_is_custom": false, + "rate_limit_admin_locked": false, + "usage_1h": 120, + "usage_24h": 2880, + "usage_7d": 20160, + "rejected_24h": 0, + "rejected_7d": 0 + } + ] + } + ], + "summary": { + "total_otlp_services": 5, + "active_services": 4, + "services_with_errors": 1, + "services_never_pushed": 0, + "total_teams": 2 + } +} +``` + **Audit actions:** `user.role_changed`, `user.deactivated`, `user.reactivated`, `team.created`, `team.updated`, `team.deleted`, `team.member_added`, `team.member_removed`, `team.member_role_changed`, `service.created`, `service.updated`, `service.deleted`, `settings.updated`, `manifest_sync`, `manifest_config.created`, `manifest_config.updated`, `manifest_config.deleted`, `drift.detected`, `drift.accepted`, `drift.dismissed`, `drift.reopened`, `drift.resolved`, `drift.bulk_accepted`, `drift.bulk_dismissed` ## 4.11 Alerts @@ -885,3 +1148,72 @@ Team-scoped drift flag review, actions, and bulk operations. All endpoints are n - Reopen only works on dismissed flags (400 otherwise) **Audit actions:** `drift.accepted`, `drift.dismissed`, `drift.reopened`, `drift.bulk_accepted`, `drift.bulk_dismissed` + +## 4.18 OTLP Receiver **[Implemented]** + +OpenTelemetry metrics push endpoint. Mounted at `/v1/metrics` (standard OTLP HTTP path), **not** under `/api/`. Authenticated via API key, not session. Mounted before session/CSRF middleware in the middleware chain. + +| Method | Path | Auth | Description | +|---|---|---|---| +| POST | `/v1/metrics` | requireApiKeyAuth | Accept OTLP JSON metrics payload. | + +**Request:** OTLP `ExportMetricsServiceRequest` JSON body (limit: 1MB). + +**Rate limiting:** Two layers: +1. **Global IP-based:** 600 requests/minute per IP (configurable via `OTLP_RATE_LIMIT_MAX` and `OTLP_RATE_LIMIT_WINDOW_MS`). Applied before API key auth. +2. **Per-key token bucket:** Configurable per API key (default 150,000 req/min via `OTLP_PER_KEY_RATE_LIMIT_RPM`). See section 9.4 in the security spec. + +**Usage tracking:** Every request increments `push_count` in minute and hour buckets. Rejected requests (429) additionally increment `rejected_count`. Data is flushed to `api_key_usage_buckets` every 5 seconds (configurable via `OTLP_USAGE_FLUSH_INTERVAL_MS`). + +**Middleware chain:** `express.json (1MB limit)` → OTLP global rate limit → `requireApiKeyAuth` → per-key rate limit → usage tracking → OTLP router. + +**Processing pipeline:** + +1. Authenticate via `Authorization: Bearer dps_...` header → resolves `teamId` and `apiKeyId` +2. Parse OTLP JSON with `OtlpParser` → extracts `service.name` from resource attributes, maps gauge metrics to dependency health fields +3. **Auto-register services:** For each `service.name` in the payload: + - Look up by `name` + `team_id` + - Not found → auto-create with `health_endpoint_format = 'otlp'`, `health_endpoint = ''`, `is_active = 1`, `poll_interval_ms = 0` + - Found but `health_endpoint_format !== 'otlp'` → include warning (does not overwrite format) +4. Upsert dependencies via `DependencyUpsertService` +5. Emit `STATUS_CHANGE` events for alert processing + +**POST /v1/metrics response (success):** + +```json +{ + "partialSuccess": { + "rejectedDataPoints": 0, + "errorMessage": "" + } +} +``` + +**OTLP metric mapping (defaults, customizable via MetricSchemaConfig):** + +| OTLP Gauge Metric | Maps To | +|---|---| +| `dependency.health.status` | `health.state` (HealthState 0-2) | +| `dependency.health.healthy` | `healthy` (0 or 1) | +| `dependency.health.latency` | `health.latency` (milliseconds) | +| `dependency.health.code` | `health.code` (HTTP status code) | +| `dependency.health.check_skipped` | `health.skipped` | + +These defaults can be overridden per-service via `MetricSchemaConfig` stored in `schema_config`. See the `MetricSchemaConfig` section in the data model spec. + +**OTLP attribute mapping (defaults, customizable via MetricSchemaConfig):** + +| Resource/Data Point Attribute | Maps To | +|---|---| +| `service.name` (resource, required) | Service name for lookup/auto-registration | +| `dependency.name` (data point, required) | Dependency name | +| `dependency.type` (data point, optional) | Dependency type | +| `dependency.impact` (data point, optional) | Impact description | +| `dependency.description` (data point, optional) | Dependency description | +| `dependency.error_message` (data point, optional) | Error message | + +Attribute defaults (except `service.name`) can be overridden per-service via the `labels` field in `MetricSchemaConfig`. + +**OTLP per-service config loading:** The OTLP receiver loads `schema_config` for each service individually when processing pushed metrics, allowing per-service metric and attribute name customization. + +**Timestamp handling:** `timeUnixNano` from data points is converted to ISO string for `lastChecked`. Falls back to `Date.now()` if missing. diff --git a/docs/spec/05-health-polling.md b/docs/spec/05-health-polling.md index 4e5a132..dbc5953 100644 --- a/docs/spec/05-health-polling.md +++ b/docs/spec/05-health-polling.md @@ -115,7 +115,226 @@ Promise coalescing for services sharing the same health endpoint URL. - Each service maintains independent circuit breaker and backoff state despite sharing the HTTP response - No cross-cycle caching — each tick triggers fresh requests -## 5.7 Dependency Parsing & Upsert +## 5.7 Format-Aware Polling **[Implemented]** + +The polling system supports multiple health endpoint formats via the `health_endpoint_format` field on each service. + +### Service Format Dispatch + +| Format | Polling Behavior | Accept Header | Response Parsing | +|---|---|---|---| +| `default` | Poll endpoint, parse JSON array | `application/json` | `DependencyParser` (proactive-deps format) | +| `schema` | Poll endpoint, parse JSON with schema mapping | `application/json` | `DependencyParser` → `SchemaMapper` | +| `prometheus` | Poll endpoint, parse text | `text/plain; version=0.0.4` | `DependencyParser` → `PrometheusParser` | +| `otlp` | **Not polled** (push-only) | N/A | Receives data via `POST /v1/metrics` and `POST /v1/traces` | + +### OTLP Service Exclusion + +OTLP services are push-only and are excluded from the polling lifecycle at multiple levels: + +- `HealthPollingService.startService()`: skips services with `health_endpoint_format === 'otlp'`, does not create a poller or emit `SERVICE_STARTED` +- `DependencyParser.parse()`: throws if `format === 'otlp'` (safety check — should never be called) +- `ServicePoller`: never instantiated for OTLP services + +### OtlpParser + +Parses OTLP `ExportMetricsServiceRequest` JSON payloads into dependency statuses. Used by the OTLP receiver (`POST /v1/metrics`), not by the polling system. + +**Public methods:** + +- `parse(request)` — parses a full OTLP request, returns `OtlpParseResult[]` (one per resource) +- `parseResourceMetrics(rm, config?)` — parses a single `ResourceMetrics` block with optional `MetricSchemaConfig` +- `extractServiceName(rm)` — extracts `service.name` from resource attributes + +**Custom metric/attribute names via MetricSchemaConfig:** The OTLP receiver loads `schema_config` per-service and passes it to `parseResourceMetrics()`. Default metric names (`dependency.health.status`, etc.) and attribute names (`dependency.name`, etc.) can be overridden per-service. See the OTLP metric mapping tables in the API reference and the `MetricSchemaConfig` section in the data model spec. + +**Healthy value comparison:** Health status is determined by comparing the `healthy` metric value against a configurable `healthy_value` (default: `1`). The `healthy_value` can be overridden per-service via the `MetricSchemaConfig`. + +**Timestamp handling:** `timeUnixNano` from OTLP data points is converted from nanoseconds to an ISO 8601 string for `lastChecked`. Falls back to `Date.now()` if the timestamp is missing or zero. + +### OTLP Receiver Endpoint + +The OTLP receiver (`POST /v1/metrics`) is documented in the API reference (section 4.18). Key implementation details relevant to the polling system: + +**Middleware chain:** `express.json (1MB limit)` → `OTLP global rate limit (600/min per IP)` → `requireApiKeyAuth` → `per-key rate limit (token bucket)` → `usage tracking` → `OTLP router`. + +**Auto-registration:** When the receiver encounters a `service.name` not yet registered for the authenticated team, it auto-creates a service with `health_endpoint_format = 'otlp'`, `health_endpoint = ''`, `is_active = 1`, `poll_interval_ms = 0`. These services are excluded from the polling lifecycle. + +**Format mismatch:** If a service exists but has a different `health_endpoint_format`, the receiver logs a warning but continues processing. It does not overwrite the existing format. + +**Response format:** Returns an OTLP-standard `partialSuccess` response. On success: `{ partialSuccess: { rejectedDataPoints: 0, errorMessage: "" } }`. Warnings from parsing are aggregated into the `errorMessage` field. On rate limit rejection (429): `{ partialSuccess: { rejectedDataPoints: 0, errorMessage: "Rate limit exceeded..." } }`. + +**Per-key rate limiting:** See section 9.4 in the security spec for details on the token bucket algorithm, configuration, and response headers. + +**Usage tracking:** Every request (accepted or rejected) increments `push_count` in dual-granularity buckets (minute + hour). Rejected requests additionally increment `rejected_count`. Usage data is accumulated in-memory and flushed to the database every `OTLP_USAGE_FLUSH_INTERVAL_MS` (default 5s). + +### Histogram and Sum Metric Processing **[Implemented]** + +The OTLP parser processes three metric types: gauges (existing), histograms (new), and sums (new). + +**Histogram processing:** When `metric.histogram?.dataPoints` is present, each data point's bucket boundaries and counts are passed to `computePercentiles()` in `server/src/utils/histogramPercentiles.ts`. This produces percentile latency (p50, p95, p99) via linear interpolation within histogram buckets. The `metric.unit` field is read for automatic unit detection (seconds → milliseconds conversion). + +**Sum processing:** When `metric.sum?.dataPoints` is present: +- Non-monotonic sums (`isMonotonic === false`) are treated as gauge values and extracted directly +- Monotonic sums store raw count as `requestCount` + +**Percentile computation algorithm:** +1. Walk cumulative bucket counts +2. Find the bucket where cumulative count crosses `count × percentile` +3. Linearly interpolate within that bucket +4. Edge cases: empty histogram (count=0) returns zeros; overflow bucket capped at last explicit bound + +**Integration with upsert pipeline:** When histogram-derived percentile data exists in `ProactiveDepsStatus.health.percentiles`, `DependencyUpsertService` calls `latencyStore.recordWithPercentiles()` with source `'otlp_histogram'` instead of the standard `latencyStore.record()`. + +**Latency bucket queries:** `getLatencyBuckets()` and `getAggregateLatencyBuckets()` include `ROUND(AVG(p50_ms))`, `ROUND(AVG(p95_ms))`, `ROUND(AVG(p99_ms))` in the SELECT for time-bucketed percentile averages. + +### Trace Ingestion **[Implemented]** + +The trace ingestion system receives OTLP trace payloads via `POST /v1/traces`, stores all spans, and automatically discovers dependencies from CLIENT and PRODUCER spans. + +```mermaid +flowchart TD + A[POST /v1/traces] --> B[Validate resourceSpans array] + B --> C{For each ResourceSpans} + C --> D[Extract service.name] + D --> E[findOrCreateService] + E --> F[SpanStore.bulkInsert — ALL spans] + F --> G[TraceParser.parseResourceSpans — CLIENT/PRODUCER only] + G --> H[TraceDependencyBridge.bridgeToDepsStatus] + H --> I[DependencyUpsertService.upsert] + I --> J[AutoAssociator.processDiscoveredDependencies] + J --> K[Emit status change events] +``` + +**Middleware chain:** `express.json (2MB limit)` → `OTLP global rate limit` → `requireApiKeyAuth` → `per-key rate limit` → `usage tracking` → `trace router`. + +**Response format:** OTLP-standard `{ partialSuccess: { rejectedDataPoints, errorMessage } }`. + +#### TraceParser + +`server/src/services/polling/TraceParser.ts` — Parses OTLP `ExportTraceServiceRequest` JSON payloads into per-service dependency results. Only CLIENT and PRODUCER spans produce dependencies (outbound calls). + +**Public API:** +- `parseRequest(data: unknown): TraceDependencyResult[]` — parses full request +- `parseResourceSpans(rs: OtlpResourceSpans): TraceDependencyResult` — parses single resource +- `extractServiceName(rs: OtlpResourceSpans): string | undefined` — reuses OtlpParser pattern + +**Target name resolution chain** (first non-empty wins): +1. `peer.service` attribute +2. `db.system` / `db.system.name` +3. `messaging.system` +4. `rpc.system` / `rpc.system.name` +5. `server.address` +6. Hostname extracted from `url.full` + +**Dependency type inference:** +| Attribute | Inferred Type | +|---|---| +| `db.system` = redis/memcached | `cache` | +| `db.system` (other) | `database` | +| `messaging.system` | `message_queue` | +| `rpc.system` = grpc | `grpc` | +| `http.request.method` | `rest` | +| (default) | `other` | + +**Auto-generated descriptions:** +- HTTP: `"{method} {host}{path}"` +- DB: `"{op} {ns}.{collection}"` +- Messaging: `"{op} {destination}"` +- gRPC: `"{rpc.method}"` + +**Deduplication:** Dependencies are deduplicated by target name within a single push — latency is averaged, error uses any-error-wins logic. + +**Output types:** +```typescript +interface TraceDependency { + targetName: string; + type: DependencyType; + latencyMs: number; + isError: boolean; + spanKind: number; + description: string; + attributes: Record; +} + +interface TraceDependencyResult { + serviceName: string; + dependencies: TraceDependency[]; +} +``` + +#### TraceDependencyBridge + +`server/src/services/polling/TraceDependencyBridge.ts` — Converts `TraceDependency[]` to `ProactiveDepsStatus[]` for the existing upsert pipeline. + +**Public API:** `bridgeToDepsStatus(traceDeps: TraceDependency[]): TraceBridgedDepsStatus[]` + +**Mapping:** +- `isError: false` → `health.state = 0` (OK), `health.code = 200` +- `isError: true` → `health.state = 2` (CRITICAL), `health.code = 500` +- Span duration → `health.latency` +- All outputs include `discovery_source: 'otlp_trace'` + +#### AutoAssociator + +`server/src/services/polling/AutoAssociator.ts` — Automatically links trace-discovered dependencies to registered services. + +**Public API:** `processDiscoveredDependencies(sourceService: Service, dependencies: ProactiveDepsStatus[], teamId: string): void` + +**Matching strategy:** +1. Case-insensitive exact name match against team services +2. Canonical name resolution via `DependencyAliasStore`, then match service name + +**Association type mapping:** +| Dependency Type | Association Type | +|---|---| +| `database` | `database` | +| `cache` | `cache` | +| `message_queue` | `message_queue` | +| `rest` or `grpc` | `api_call` | +| (default) | `other` | + +**Safety rules:** +- Skips self-links (source service = target service) +- Skips already-associated pairs (including dismissed — never re-suggests) +- Catches UNIQUE constraint violations as no-ops (race condition safety) +- Creates associations with `is_auto_suggested: true` + +#### Shared Service Resolution + +`server/src/services/polling/otlpServiceResolver.ts` — Shared `findOrCreateService()` helper used by both `/v1/metrics` and `/v1/traces` routes. Extracted from the metrics route for reuse. Auto-creates services with `health_endpoint_format = 'otlp'`, `health_endpoint = ''`, `is_active = 1`, `poll_interval_ms = 0`. + +### PrometheusParser + +Parses Prometheus text exposition format (`metric_name{labels} value`) into `ProactiveDepsStatus[]`. + +**Default metric mapping (customizable via MetricSchemaConfig):** + +| Prometheus Metric | Maps To | Notes | +|---|---|---| +| `dependency_health_status` | `health.state` | HealthState 0-2 | +| `dependency_health_healthy` | `healthy` | 0 or 1 | +| `dependency_health_latency_ms` | `health.latency` | Milliseconds (default). Set `latency_unit: 's'` to convert seconds → ms. | +| `dependency_health_code` | `health.code` | HTTP status code | +| `dependency_health_check_skipped` | `health.skipped` | | + +**Default label mapping (customizable via MetricSchemaConfig):** `name` (required), `type`, `impact`, `description`, `error_message` (all optional). + +**Custom metric/label names via MetricSchemaConfig:** + +When a service has a `MetricSchemaConfig` in its `schema_config` column, the parser merges user-provided metric and label mappings into the defaults. For example, if a user maps `{ "my_latency_metric": "latency" }`, the default `dependency_health_latency_ms` entry is removed and `my_latency_metric` is used instead. See the `MetricSchemaConfig` section in the data model spec. + +**Latency handling:** Latency is treated as milliseconds by default (no conversion). When `latency_unit` is set to `'s'` in the `MetricSchemaConfig`, the raw value is multiplied by 1000 to convert seconds to milliseconds. + +**Parsing rules:** +- `# HELP` and `# TYPE` comment lines are skipped +- Lines parsed as `metric_name{label1="val1",label2="val2"} value` +- Dependencies are grouped by `name` label and metrics are merged per dependency +- Missing `name` label produces a warning, line is skipped +- Unknown metric names are silently ignored +- `lastChecked` defaults to current time (no timestamp in text format) + +## 5.8 Dependency Parsing & Upsert When a poll succeeds, the health endpoint response is parsed (proactive-deps format) and each dependency is upserted: @@ -141,7 +360,7 @@ The `DependencyParser.parseItem()` method extracts the following optional fields Both fields follow the same pattern: present and valid → included in `ProactiveDepsStatus`; missing or invalid type → `undefined`. -## 5.8 Events +## 5.9 Events | Event | Emitted When | Payload | |---|---|---| @@ -153,7 +372,7 @@ Both fields follow the same pattern: present and valid → included in `Proactiv | `circuit:open` | Circuit transitions to open | serviceId, serviceName | | `circuit:close` | Circuit closes from half-open | serviceId, serviceName | -## 5.9 Constants Summary +## 5.10 Constants Summary | Constant | Value | Location | |---|---|---| diff --git a/docs/spec/06-dependency-graph.md b/docs/spec/06-dependency-graph.md index 5c05cd5..d328fb9 100644 --- a/docs/spec/06-dependency-graph.md +++ b/docs/spec/06-dependency-graph.md @@ -23,6 +23,7 @@ The `GraphService` builds graphs by querying services and dependencies, then con - Grouped by normalized dependency name (lowercase + trim) - ID: `external-{SHA256(normalizedName)[0:12]}` - Multiple dependencies with the same name produce a single external node with aggregated health stats +- External nodes can have org-wide enrichment applied from the `external_node_enrichment` table (display name, description, impact, contact, service type override). `GraphService` loads enrichments and applies them via `ExternalNodeBuilder.applyEnrichment()` during graph building. ## 6.3 Service Type Inference @@ -36,11 +37,48 @@ Edges represent "depends on" relationships. For each dependency: - If the dependency has an association (`target_service_id`), the edge connects the associated service to the owning service - If unassociated, the edge connects the external node to the owning service - Edge data includes `skipped: boolean` — true when the underlying dependency has `skipped = 1`. Skipped edges represent dependencies whose health checks are intentionally not executed. +- Edge data includes `discoverySource?: DiscoverySource` — indicates how the dependency was created (`'manual'`, `'otlp_metric'`, or `'otlp_trace'`) +- Edge data includes `isAutoSuggested?: boolean` — true when the association was automatically created from trace data and has not been confirmed by a user +- Edge data includes `associationId?: string | null` — the association ID, used by confirm/dismiss UI actions - Edge ID format: `{sourceId}-{depId}-{type}` — prevents duplicate edges for the same source→target→type combination +**Discovery source styling (frontend):** +- **Auto-discovered unconfirmed** (`discoverySource === 'otlp_trace'` and `isAutoSuggested === true`): rendered with dashed line and "Suggested" badge +- **Confirmed or manual**: rendered with solid line (standard treatment) + ## 6.5 Upstream Traversal The service subgraph uses `traverseUpstream()` to recursively follow dependency associations: 1. Start with the selected service 2. For each dependency with an association, add the associated service and recurse 3. Track visited services to prevent infinite loops from circular dependencies + +## 6.6 Discovery Source Integration **[Implemented]** + +The graph surfaces trace-discovered dependencies with visual distinction from manually-configured ones. + +### Graph Edge Data Extensions + +`GraphEdgeData` includes: +- `discoverySource?: DiscoverySource` — populated from `dependencies.discovery_source` in `DependencyGraphBuilder.createEdgeData()` +- `isAutoSuggested?: boolean` — populated from `dependency_associations.is_auto_suggested` (true when `is_auto_suggested === 1`) +- `associationId?: string | null` — the association ID for confirm/dismiss actions + +Service nodes include: +- `discoveredDependencyCount?: number` — count of trace-discovered dependencies for the service + +### Edge Details Panel + +The `EdgeDetailsPanel` shows: +- Discovery source badge (manual, OTLP metric, OTLP trace) +- User-enriched name/description/impact with fallback to auto-detected values +- Display name resolution: `user_display_name` → `canonicalName` → `linked_service.name` → `name` +- "Confirm" / "Dismiss" buttons for auto-suggested associations (calls `PUT /api/dependencies/:depId/associations/:assocId/confirm` or `/dismiss`) + +### Node Details Panel — External Node Enrichment + +When a node `isExternal`, the `NodeDetailsPanel` shows enriched metadata (display name, description, impact, contact) if available from the `external_node_enrichment` table. + +### Dependency Queries + +`DependencyStore.findAllWithAssociationsAndLatency` and `findByServiceIdsWithAssociationsAndLatency` SELECT `d.discovery_source` and `da.is_auto_suggested`, which flow through `DependencyGraphBuilder.createEdgeData()` into the graph response. diff --git a/docs/spec/09-security.md b/docs/spec/09-security.md index 2e18dc3..00a63d3 100644 --- a/docs/spec/09-security.md +++ b/docs/spec/09-security.md @@ -48,13 +48,43 @@ When no cert paths are provided, the server generates a self-signed certificate ## 9.4 Rate Limiting **[Implemented]** +### IP-Based Rate Limiters + | Limiter | Window | Max Requests | Scope | |---|---|---|---| | Global | 1 minute | 3,000 per IP | All requests (applied before session middleware) | | Auth | 1 minute | 20 per IP | `/api/auth` endpoints only | +| OTLP Global | 1 minute | 600 per IP | `POST /v1/metrics` only (applied before API key auth) | Returns `429 Too Many Requests` with `RateLimit-*` and `Retry-After` headers. +### Per-Key Rate Limiting **[Implemented]** + +In addition to the IP-based OTLP global limiter, each API key is individually rate-limited via a **token bucket** algorithm. + +**Algorithm:** Token bucket with continuous refill. +- **Bucket capacity:** `(effective_rpm / 60) × OTLP_RATE_LIMIT_BURST_SECONDS` tokens +- **Refill rate:** `effective_rpm / 60 / 1000` tokens per millisecond +- **Lazy initialization:** Buckets are created on first request per key +- **Eviction:** Buckets are evicted when a key's rate limit is changed (via admin or team endpoint), forcing re-initialization on next request + +**Rate limit resolution hierarchy:** +1. `rate_limit_rpm = 0` → Unlimited (admin-only) — rate limiting skipped entirely +2. `rate_limit_rpm = N` (non-null) → Custom limit of N requests/minute +3. `rate_limit_rpm = NULL` → System default from `OTLP_PER_KEY_RATE_LIMIT_RPM` env var (default: 150,000) + +**On rejection (429):** Returns OTLP-compatible response `{ partialSuccess: { rejectedDataPoints: 0, errorMessage: "..." } }` with `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `Retry-After`, and `X-RateLimit-Key` headers. Rejected requests are counted in usage tracking (`rejected_count`). + +**Soft limit warning:** When usage exceeds `OTLP_RATE_LIMIT_WARNING_THRESHOLD` (default: 80%) of bucket capacity, the response includes an `X-RateLimit-Warning: true` header. A server-side warning is logged at most once per 15 minutes per key. + +**Development bypass:** Per-key rate limiting is skipped entirely when `NODE_ENV === 'development'`. + +| Environment Variable | Default | Description | +|---|---|---| +| `OTLP_PER_KEY_RATE_LIMIT_RPM` | `150000` | Default per-key rate limit (requests/minute) when `rate_limit_rpm` is NULL | +| `OTLP_RATE_LIMIT_BURST_SECONDS` | `6` | Burst window; bucket capacity = refill rate × burst seconds | +| `OTLP_RATE_LIMIT_WARNING_THRESHOLD` | `0.80` | Fraction of capacity consumed before warning header is set | + ## 9.5 Redirect Validation **[Implemented]** Logout redirect URLs are validated client-side via `validateRedirectUrl()`: @@ -74,7 +104,8 @@ The order matters — each layer builds on previous: | 3 | HTTPS Redirect | 301 redirect if enabled | | 4 | CORS | `credentials: true`, configurable origin | | 5 | JSON Parser | `express.json()` body parsing | -| 6 | Global Rate Limit | 100 req/15min per IP — early rejection before session creation | +| 5.5 | OTLP Route | `POST /v1/metrics` — JSON (1MB limit), OTLP global rate limit (600/min per IP), API key auth, per-key rate limit (token bucket), usage tracking, OTLP router. Mounted before session/CSRF. | +| 6 | Global Rate Limit | 3,000 req/min per IP — early rejection before session creation | | 7 | Session | Populates `req.session` | | 8 | Auth Bypass | Dev-only auto-auth | | 9 | Request Logger | `pino-http` structured logging (method, path, status, response time, userId) | diff --git a/docs/spec/11-configuration.md b/docs/spec/11-configuration.md index bdf7791..e2312ff 100644 --- a/docs/spec/11-configuration.md +++ b/docs/spec/11-configuration.md @@ -44,6 +44,12 @@ All configuration is via environment variables on the server (set in `server/.en | `RATE_LIMIT_MAX` | `3000` | Max requests per IP per global window | | `AUTH_RATE_LIMIT_WINDOW_MS` | `60000` (1 min) | Auth endpoint rate limit window | | `AUTH_RATE_LIMIT_MAX` | `20` | Max auth requests per IP per window | +| `OTLP_RATE_LIMIT_WINDOW_MS` | `60000` (1 min) | OTLP receiver global rate limit window (IP-based) | +| `OTLP_RATE_LIMIT_MAX` | `600` | Max OTLP requests per IP per global window | +| `OTLP_PER_KEY_RATE_LIMIT_RPM` | `150000` | Default per-key rate limit (requests/minute) when key's `rate_limit_rpm` is NULL | +| `OTLP_RATE_LIMIT_BURST_SECONDS` | `6` | Per-key token bucket burst window; capacity = (rpm/60) × this value | +| `OTLP_RATE_LIMIT_WARNING_THRESHOLD` | `0.80` | Fraction of per-key bucket capacity consumed before `X-RateLimit-Warning` header is set | +| `OTLP_USAGE_FLUSH_INTERVAL_MS` | `5000` (5s) | How often the in-memory usage accumulator flushes to the database | ## 11.5 Polling diff --git a/docs/spec/13-store-layer.md b/docs/spec/13-store-layer.md index eb33dd8..4dd712f 100644 --- a/docs/spec/13-store-layer.md +++ b/docs/spec/13-store-layer.md @@ -24,6 +24,11 @@ class StoreRegistry { public readonly manifestConfig: IManifestConfigStore; public readonly manifestSyncHistory: IManifestSyncHistoryStore; public readonly driftFlags: IDriftFlagStore; + public readonly teamApiKeys: ITeamApiKeyStore; + public readonly apiKeyUsage: IApiKeyUsageStore; + public readonly spans: ISpanStore; + public readonly appSettings: IAppSettingsStore; + public readonly externalNodeEnrichment: IExternalNodeEnrichmentStore; static getInstance(): StoreRegistry; // Singleton for production static create(database): StoreRegistry; // Scoped instance for testing @@ -101,12 +106,18 @@ findExistingByServiceId(serviceId: string): ExistingDependency[] findDependentReports(serviceId: string): DependentReport[] upsert(input: DependencyUpsertInput): UpsertResult updateOverrides(id: string, overrides: DependencyOverrideInput): Dependency | undefined +updateUserEnrichment(id: string, enrichment: DependencyUserEnrichmentInput): Dependency | undefined +findByDiscoverySource(serviceId: string, source: string): Dependency[] delete(id: string): boolean deleteByServiceId(serviceId: string): number exists(id: string): boolean count(options?: DependencyListOptions): number ``` +`DependencyUserEnrichmentInput`: `{ user_display_name?: string | null; user_description?: string | null; user_impact?: string | null }`. Targeted UPDATE that only touches user enrichment columns and `updated_at`. Returns `undefined` if dependency not found. + +`findByDiscoverySource(serviceId, source)`: Returns all dependencies for a service filtered by `discovery_source`. Used by the discovered dependencies list endpoint. + `DependencyOverrideInput`: `{ contact_override?: string | null; impact_override?: string | null }`. Targeted UPDATE that only touches `contact_override`, `impact_override`, and `updated_at` — does not interfere with polled data columns. Returns `undefined` if dependency not found. Passing a key with `null` clears that override; omitting a key leaves it unchanged. `DependencyWithResolvedOverrides`: Extends `Dependency` with `effective_contact: string | null` and `effective_impact: string | null`. Computed at the API layer by `resolveDependencyOverrides(dependencies, teamId?)` in `server/src/utils/dependencyOverrideResolver.ts`, not stored in the database. Used in service detail and list API responses. Uses a 4-tier override hierarchy: instance override > team canonical override > global canonical override > polled data. When `teamId` is provided, team-scoped overrides take precedence over global ones. @@ -117,24 +128,49 @@ findById(id: string): DependencyAssociation | undefined findByDependencyId(dependencyId: string): DependencyAssociation[] findByDependencyIdWithService(dependencyId: string): AssociationWithService[] findByLinkedServiceId(linkedServiceId: string): DependencyAssociation[] +findAutoSuggested(dependencyId: string): DependencyAssociation[] existsForDependencyAndService(dependencyId: string, linkedServiceId: string): boolean create(input: AssociationCreateInput): DependencyAssociation +confirm(id: string): boolean +dismiss(id: string): boolean delete(id: string): boolean deleteByDependencyId(dependencyId: string): number +deleteOldDismissed(olderThan: string): number exists(id: string): boolean count(options?: AssociationListOptions): number ``` +`findAutoSuggested(dependencyId)`: Returns associations where `is_auto_suggested = 1` and `is_dismissed = 0`. + +`confirm(id)`: Sets `is_auto_suggested = 0` (promotes to confirmed). Returns false if not found. + +`dismiss(id)`: Sets `is_dismissed = 1`. Returns false if not found. Dismissed associations are never re-suggested. + +`deleteOldDismissed(olderThan)`: Deletes associations where `is_dismissed = 1 AND created_at < ?`. Used by `DataRetentionService`. + +`AssociationCreateInput`: extended with optional `is_auto_suggested?: boolean` for trace-discovered associations. + ### ILatencyHistoryStore ```typescript record(dependencyId: string, latencyMs: number, timestamp: string): DependencyLatencyHistory +recordWithPercentiles(dependencyId: string, latencyMs: number, percentiles: PercentileInput, timestamp: string, source: string): DependencyLatencyHistory getStats24h(dependencyId: string): LatencyStats getAvgLatency24h(dependencyId: string): number | null getHistory(dependencyId: string, options?: { startTime?: string; endTime?: string; limit?: number }): LatencyDataPoint[] +getLatencyBuckets(dependencyId: string, range: LatencyRange): LatencyBucket[] +getAggregateLatencyBuckets(dependencyIds: string[], range: LatencyRange): LatencyBucket[] deleteOlderThan(timestamp: string): number deleteByDependencyId(dependencyId: string): number ``` +`PercentileInput`: `{ p50?: number; p95?: number; p99?: number; min?: number; max?: number; requestCount?: number }`. + +`recordWithPercentiles`: Stores a latency data point with percentile breakdown. `source` indicates origin (e.g., `'otlp_histogram'`). Used when histogram data is available from OTLP metric pushes. + +`LatencyBucket`: `{ timestamp, min, avg, max, count, avg_p50?, avg_p95?, avg_p99? }`. Time-bucketed aggregations with optional averaged percentiles. + +`LatencyRange`: `'1h' | '6h' | '24h' | '7d' | '30d'`. Determines bucket granularity (1min for 1h, 5min for 6h, 30min for 24h, 6h for 7d, 1d for 30d). + ### IErrorHistoryStore ```typescript record(dependencyId: string, error: string | null, errorMessage: string | null, timestamp: string): DependencyErrorHistory @@ -319,3 +355,96 @@ deleteOlderThan(timestamp: string, statuses?: DriftFlagStatus[]): number - `upsertRemovalDrift`: pending or dismissed exists → update last_detected_at (stay in current status); not found → create new `deleteOlderThan` supports optional status filter array for targeted cleanup (e.g., only delete terminal statuses like `accepted`, `resolved`). + +### ITeamApiKeyStore **[Implemented]** +```typescript +findByTeamId(teamId: string): TeamApiKey[] +findByKeyHash(hash: string): TeamApiKey | undefined +findById(id: string): TeamApiKey | undefined +create(input: CreateTeamApiKeyInput): TeamApiKey & { rawKey: string } +delete(id: string): boolean +updateLastUsed(id: string): void +updateRateLimit(id: string, rateLimit: number | null): TeamApiKey +setAdminLock(id: string, locked: boolean, rateLimit?: number | null): TeamApiKey +``` + +`CreateTeamApiKeyInput`: `{ team_id: string; name: string; created_by?: string }`. Generates a UUID `id`, raw key (`dps_` + 16 random hex bytes), SHA-256 hash, and 8-character prefix. Returns the full `TeamApiKey` record plus `rawKey` (shown once, never stored). + +`findByTeamId` returns keys sorted by `created_at DESC`. + +`findByKeyHash` is the primary lookup used during authentication — indexed for fast access. + +`findById` is used by the per-key rate limiter to look up the key's effective rate limit and by the rate limit management endpoints. + +`updateLastUsed` sets `last_used_at` to current timestamp. Called asynchronously during API key auth (non-critical failure). + +`updateRateLimit` sets `rate_limit_rpm` to the given value (positive integer, null for system default). Returns the updated `TeamApiKey`. Used by both team-level and admin rate limit endpoints. + +`setAdminLock` sets `rate_limit_admin_locked` (0 or 1) and optionally updates `rate_limit_rpm` in the same operation (uses SQL `COALESCE` to skip rate limit update when not provided). Returns the updated `TeamApiKey`. Admin-only. + +### IApiKeyUsageStore **[Implemented]** +```typescript +bulkUpsert(entries: BulkUpsertEntry[]): void +getBuckets(apiKeyId: string, granularity: 'minute' | 'hour', from: string, to: string): ApiKeyUsageBucket[] +getBucketsByTeam(teamId: string, granularity: 'minute' | 'hour', from: string, to: string): (ApiKeyUsageBucket & { key_name: string; key_prefix: string })[] +getAllBuckets(granularity: 'minute' | 'hour', from: string, to: string): (ApiKeyUsageBucket & { team_id: string; key_name: string })[] +getSummaryForKeys(apiKeyIds: string[], from: string, to: string): Map +pruneMinuteBuckets(olderThan: string): number +pruneHourBuckets(olderThan: string): number +pruneOrphanedBuckets(olderThan: string): number +``` + +`BulkUpsertEntry`: `{ api_key_id: string; bucket_start: string; granularity: 'minute' | 'hour'; push_count: number; rejected_count: number }`. Uses `INSERT ... ON CONFLICT DO UPDATE SET push_count = push_count + excluded.push_count, rejected_count = rejected_count + excluded.rejected_count` for atomic accumulation. + +`getBuckets` returns time-series buckets for a single key, ordered by `bucket_start ASC`. + +`getBucketsByTeam` joins through `team_api_keys` to return buckets for all keys belonging to a team, enriched with key metadata. + +`getAllBuckets` returns buckets across all keys (admin dashboard), enriched with `team_id` and `key_name`. + +`getSummaryForKeys` returns aggregated `push_count` and `rejected_count` totals for a set of key IDs within a time range. Used by the OTLP stats endpoints for usage summaries (1h, 24h, 7d windows). + +**Retention pruning:** +- `pruneMinuteBuckets(olderThan)` — deletes minute-granularity buckets older than the given timestamp (24h retention) +- `pruneHourBuckets(olderThan)` — deletes hour-granularity buckets older than the given timestamp (30d retention) +- `pruneOrphanedBuckets(olderThan)` — deletes buckets where `api_key_id` no longer exists in `team_api_keys` and `bucket_start` is older than the given timestamp (7d grace period) + +### ISpanStore **[Implemented]** +```typescript +bulkInsert(spans: CreateSpanInput[]): number +findByTraceId(traceId: string): Span[] +findByServiceName(serviceName: string, options?: { since?: string; limit?: number }): Span[] +deleteOlderThan(timestamp: string): number +``` + +`bulkInsert`: Inserts multiple spans in a transaction using a prepared statement for batch performance. Generates UUIDs for each span. Returns the number of spans inserted. + +`findByTraceId`: Returns all spans for a trace ordered by `start_time ASC`. Used for trace timeline reconstruction. + +`findByServiceName`: Returns spans filtered by service name, optionally filtered by `since` timestamp. Default limit 1000, ordered by `start_time DESC`. + +`deleteOlderThan`: Deletes spans where `created_at < timestamp`. Used by `DataRetentionService` for configurable span retention cleanup. + +### IAppSettingsStore **[Implemented]** +```typescript +get(key: string): string | undefined +set(key: string, value: string, updatedBy?: string): void +``` + +Simple key-value store for admin-configurable settings. `get` returns `undefined` for missing keys. `set` uses `INSERT OR REPLACE` (upsert semantics). Used by `DataRetentionService` to read `span_retention_days` and by the admin settings endpoints. + +### IExternalNodeEnrichmentStore **[Implemented]** +```typescript +findByCanonicalName(name: string): ExternalNodeEnrichment | undefined +findAll(): ExternalNodeEnrichment[] +upsert(input: UpsertExternalNodeEnrichmentInput): ExternalNodeEnrichment +delete(id: string): boolean +``` + +`findByCanonicalName`: Exact match lookup by canonical name. + +`findAll`: Returns all enrichment records ordered by `canonical_name`. + +`upsert`: Uses `INSERT ... ON CONFLICT(canonical_name) DO UPDATE` for idempotent creates/updates. Returns the created or updated record. + +`delete`: Hard deletes by ID. Returns false if not found. diff --git a/docs/spec/index.md b/docs/spec/index.md index ac08d48..f8c8d8d 100644 --- a/docs/spec/index.md +++ b/docs/spec/index.md @@ -9,11 +9,11 @@ | # | File | Topics | Keywords | |---|---|---|---| | 1 | [01-architecture.md](./01-architecture.md) | Monorepo layout, runtime topology, request flow, key design decisions | architecture, monorepo, Express, Vite, SQLite, topology, proxy, request flow | -| 2 | [02-data-model.md](./02-data-model.md) | Database config, all table definitions, type enums, migration history | database, schema, tables, columns, migrations, SQLite, ERD, types, enums, foreign keys | +| 2 | [02-data-model.md](./02-data-model.md) | Database config, all table definitions, type enums, migration history, spans, app settings, external node enrichment | database, schema, tables, columns, migrations, SQLite, ERD, types, enums, foreign keys, spans, discovery_source, percentiles, retention | | 3 | [03-auth.md](./03-auth.md) | OIDC flow, local auth, sessions, CSRF, RBAC, middleware | authentication, authorization, OIDC, PKCE, login, logout, session, CSRF, roles, middleware, local auth, passwords | | 4 | [04-api-reference.md](./04-api-reference.md) | All REST API endpoints, request/response shapes, validation rules | API, endpoints, routes, REST, request, response, CRUD, services, teams, users, aliases, associations, graph, latency, errors, admin, alerts, wallboard, overrides | -| 5 | [05-health-polling.md](./05-health-polling.md) | Polling lifecycle, circuit breaker, backoff, TTL cache, host rate limiter, deduplication, dependency parsing, events | polling, health check, circuit breaker, backoff, TTL, cache, rate limiter, deduplication, dependency parsing, events, upsert | -| 6 | [06-dependency-graph.md](./06-dependency-graph.md) | Graph building, node types, edge construction, upstream traversal | graph, nodes, edges, external nodes, traversal, subgraph, React Flow | +| 5 | [05-health-polling.md](./05-health-polling.md) | Polling lifecycle, circuit breaker, backoff, TTL cache, host rate limiter, deduplication, dependency parsing, events, format-aware dispatch, Prometheus parsing, OTLP exclusion, trace ingestion, histogram/sum processing, auto-association | polling, health check, circuit breaker, backoff, TTL, cache, rate limiter, deduplication, dependency parsing, events, upsert, prometheus, otlp, format, traces, histogram, percentiles, auto-association | +| 6 | [06-dependency-graph.md](./06-dependency-graph.md) | Graph building, node types, edge construction, upstream traversal, discovery source styling, external node enrichment | graph, nodes, edges, external nodes, traversal, subgraph, React Flow, discovery source, auto-suggested, enrichment | | 8 | [08-ssrf.md](./08-ssrf.md) | Blocked IP ranges, two-step validation, allowlist | SSRF, security, IP ranges, DNS rebinding, allowlist, private networks | | 9 | [09-security.md](./09-security.md) | Security headers, HTTPS redirect, rate limiting, redirect validation, middleware order | security, headers, Helmet, CSP, HTTPS, rate limiting, middleware order | | 10 | [10-client-architecture.md](./10-client-architecture.md) | Routing, context providers, API client pattern, custom hooks, localStorage keys | client, React, routing, AuthContext, ThemeContext, hooks, localStorage, API client | @@ -35,6 +35,8 @@ When working on a task, use these mappings to find the right sections: - **Alert system changes** → 12 (Planned Features §12.6) + 04 (API Reference §4.11) - **Frontend component changes** → 10 (Client Architecture) + 04 (API Reference) - **Configuration/env var changes** → 11 (Configuration) +- **OTLP / Prometheus ingestion** → 04 (API Reference §4.18) + 05 (Health Polling §5.7) + 03 (Auth §3.8) + 02 (Data Model: team_api_keys) +- **API key management** → 03 (Auth §3.8) + 04 (API Reference §4.4 Team API Keys) + 13 (Store Layer: ITeamApiKeyStore) - **Schema mapping / custom health endpoints** → 12 (Planned Features §12.5) + 05 (Health Polling) - **Manifest sync / drift detection** → 15 (Manifest Sync) + 13 (Store Layer) + 08 (SSRF) + [Manifest Schema Reference](../manifest-schema.md) - **Deployment / Docker** → 12 (Planned Features §12.9) + 11 (Configuration) diff --git a/server/.env.example b/server/.env.example index 8cb830b..f422fbb 100644 --- a/server/.env.example +++ b/server/.env.example @@ -60,6 +60,19 @@ RATE_LIMIT_MAX=3000 # Auth rate limit applies to /api/auth endpoints (default: 20 requests per 1 minute) AUTH_RATE_LIMIT_WINDOW_MS=60000 AUTH_RATE_LIMIT_MAX=20 +# OTLP global rate limit applies to POST /v1/metrics per IP (default: 600 requests per 1 minute) +# Higher than global limit to accommodate collector traffic +OTLP_RATE_LIMIT_WINDOW_MS=60000 +OTLP_RATE_LIMIT_MAX=600 +# OTLP per-key rate limiting (token bucket per API key) +# Default per-key limit when key's rate_limit_rpm is NULL (default: 150000) +# OTLP_PER_KEY_RATE_LIMIT_RPM=150000 +# Burst window in seconds; bucket capacity = (rpm/60) × burst seconds (default: 6) +# OTLP_RATE_LIMIT_BURST_SECONDS=6 +# Fraction of bucket capacity consumed before X-RateLimit-Warning header is set (default: 0.80) +# OTLP_RATE_LIMIT_WARNING_THRESHOLD=0.80 +# How often to flush the in-memory usage accumulator to the database in ms (default: 5000) +# OTLP_USAGE_FLUSH_INTERVAL_MS=5000 # Logging # Log level: fatal, error, warn, info, debug, trace, silent (default: info) diff --git a/server/.eslintrc.json b/server/.eslintrc.json index de2c733..3f04305 100644 --- a/server/.eslintrc.json +++ b/server/.eslintrc.json @@ -16,6 +16,6 @@ }, "plugins": ["@typescript-eslint", "security"], "rules": { - "@typescript-eslint/no-unused-vars": ["error", { "argsIgnorePattern": "^_" }] + "@typescript-eslint/no-unused-vars": ["error", { "argsIgnorePattern": "^_", "ignoreRestSiblings": true }] } } diff --git a/server/src/auth/apiKeyAuth.test.ts b/server/src/auth/apiKeyAuth.test.ts new file mode 100644 index 0000000..7212dcd --- /dev/null +++ b/server/src/auth/apiKeyAuth.test.ts @@ -0,0 +1,153 @@ +/* eslint-disable @typescript-eslint/no-explicit-any */ +import { Request, Response } from 'express'; +import Database from 'better-sqlite3'; +import { createHash, randomUUID } from 'crypto'; + +// Create in-memory database for testing +const testDb = new Database(':memory:'); + +// Mock the db module +jest.mock('../db', () => ({ + db: testDb, + default: testDb, +})); + +import { requireApiKeyAuth } from './apiKeyAuth'; + +describe('requireApiKeyAuth', () => { + const teamId = randomUUID(); + const rawKey = 'dps_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4'; + const keyHash = createHash('sha256').update(rawKey).digest('hex'); + const keyId = randomUUID(); + + const createMockRequest = (overrides: Partial = {}): Request => { + return { + headers: {}, + params: {}, + body: {}, + ...overrides, + } as Request; + }; + + const createMockResponse = (): Response => { + const res: Partial = { + status: jest.fn().mockReturnThis(), + json: jest.fn().mockReturnThis(), + }; + return res as Response; + }; + + beforeAll(() => { + testDb.pragma('foreign_keys = OFF'); + + testDb.exec(` + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + `); + + testDb + .prepare( + `INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix) + VALUES (?, ?, ?, ?, ?)`, + ) + .run(keyId, teamId, 'Test Key', keyHash, rawKey.slice(0, 8)); + }); + + afterAll(() => { + testDb.close(); + }); + + it('should authenticate with a valid API key and set apiKeyTeamId', () => { + const req = createMockRequest({ + headers: { authorization: `Bearer ${rawKey}` }, + } as any); + const res = createMockResponse(); + const next = jest.fn(); + + requireApiKeyAuth(req, res, next); + + expect(next).toHaveBeenCalled(); + expect(req.apiKeyTeamId).toBe(teamId); + expect(res.status).not.toHaveBeenCalled(); + }); + + it('should return 401 when Authorization header is missing', () => { + const req = createMockRequest(); + const res = createMockResponse(); + const next = jest.fn(); + + requireApiKeyAuth(req, res, next); + + expect(next).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(401); + expect(res.json).toHaveBeenCalledWith({ error: 'Missing Authorization header' }); + }); + + it('should return 401 for malformed Authorization header (no Bearer)', () => { + const req = createMockRequest({ + headers: { authorization: `Basic ${rawKey}` }, + } as any); + const res = createMockResponse(); + const next = jest.fn(); + + requireApiKeyAuth(req, res, next); + + expect(next).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(401); + expect(res.json).toHaveBeenCalledWith({ error: 'Invalid Authorization header format' }); + }); + + it('should return 401 for key not starting with dps_', () => { + const req = createMockRequest({ + headers: { authorization: 'Bearer some_random_key' }, + } as any); + const res = createMockResponse(); + const next = jest.fn(); + + requireApiKeyAuth(req, res, next); + + expect(next).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(401); + expect(res.json).toHaveBeenCalledWith({ error: 'Invalid Authorization header format' }); + }); + + it('should return 401 for invalid API key (not in DB)', () => { + const req = createMockRequest({ + headers: { authorization: 'Bearer dps_00000000000000000000000000000000' }, + } as any); + const res = createMockResponse(); + const next = jest.fn(); + + requireApiKeyAuth(req, res, next); + + expect(next).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(401); + expect(res.json).toHaveBeenCalledWith({ error: 'Invalid API key' }); + }); + + it('should update last_used_at on successful authentication', () => { + const req = createMockRequest({ + headers: { authorization: `Bearer ${rawKey}` }, + } as any); + const res = createMockResponse(); + const next = jest.fn(); + + requireApiKeyAuth(req, res, next); + + expect(next).toHaveBeenCalled(); + + const record = testDb + .prepare('SELECT last_used_at FROM team_api_keys WHERE id = ?') + .get(keyId) as { last_used_at: string | null }; + expect(record.last_used_at).not.toBeNull(); + }); +}); diff --git a/server/src/auth/apiKeyAuth.ts b/server/src/auth/apiKeyAuth.ts new file mode 100644 index 0000000..d1007a2 --- /dev/null +++ b/server/src/auth/apiKeyAuth.ts @@ -0,0 +1,58 @@ +import { Request, Response, NextFunction } from 'express'; +import { createHash } from 'crypto'; +import { getStores } from '../stores'; + +// Extend Express Request type for API key auth +declare global { + // eslint-disable-next-line @typescript-eslint/no-namespace + namespace Express { + interface Request { + apiKeyTeamId?: string; + apiKeyId?: string; + } + } +} + +/** + * Middleware: authenticate requests via API key in Authorization header. + * Expects `Authorization: Bearer dps_...` format. + * Sets `req.apiKeyTeamId` on success, returns 401 if invalid. + * Bypasses CSRF since collectors won't have CSRF tokens. + */ +export function requireApiKeyAuth(req: Request, res: Response, next: NextFunction): void { + const authHeader = req.headers.authorization; + + if (!authHeader) { + res.status(401).json({ error: 'Missing Authorization header' }); + return; + } + + const parts = authHeader.split(' '); + if (parts.length !== 2 || parts[0] !== 'Bearer' || !parts[1].startsWith('dps_')) { + res.status(401).json({ error: 'Invalid Authorization header format' }); + return; + } + + const rawKey = parts[1]; + const keyHash = createHash('sha256').update(rawKey).digest('hex'); + + const stores = getStores(); + const apiKey = stores.teamApiKeys.findByKeyHash(keyHash); + + if (!apiKey) { + res.status(401).json({ error: 'Invalid API key' }); + return; + } + + req.apiKeyTeamId = apiKey.team_id; + req.apiKeyId = apiKey.id; + + // Update last_used_at asynchronously (fire and forget) + try { + stores.teamApiKeys.updateLastUsed(apiKey.id); + } catch { + // Non-critical — don't fail the request + } + + next(); +} diff --git a/server/src/auth/session.ts b/server/src/auth/session.ts index 0904c9c..dc2fa20 100644 --- a/server/src/auth/session.ts +++ b/server/src/auth/session.ts @@ -27,6 +27,7 @@ export const sessionMiddleware = session({ secret: validateSessionSecret(), resave: false, saveUninitialized: false, + rolling: true, cookie: { secure: 'auto', httpOnly: true, diff --git a/server/src/db/migrate.ts b/server/src/db/migrate.ts index e8a8e3b..53606c0 100644 --- a/server/src/db/migrate.ts +++ b/server/src/db/migrate.ts @@ -33,6 +33,14 @@ import * as migration030 from './migrations/030_add_alert_delay'; import * as migration031 from './migrations/031_add_alert_mutes'; import * as migration032 from './migrations/032_add_service_mutes'; import * as migration033 from './migrations/033_multi_manifest'; +import * as migration034 from './migrations/034_add_otel_sources'; +import * as migration035 from './migrations/035_api_key_rate_limit_columns'; +import * as migration036 from './migrations/036_api_key_usage_buckets'; +import * as migration037 from './migrations/037_add_trace_discovery'; +import * as migration038 from './migrations/038_add_external_node_enrichment'; +import * as migration039 from './migrations/039_add_percentile_latency'; +import * as migration040 from './migrations/040_add_span_storage'; +import * as migration041 from './migrations/041_add_span_retention_setting'; interface Migration { id: string; @@ -239,6 +247,54 @@ const migrations: Migration[] = [ name: 'multi_manifest', up: migration033.up, down: migration033.down + }, + { + id: '034', + name: 'add_otel_sources', + up: migration034.up, + down: migration034.down + }, + { + id: '035', + name: 'api_key_rate_limit_columns', + up: migration035.up, + down: migration035.down + }, + { + id: '036', + name: 'api_key_usage_buckets', + up: migration036.up, + down: migration036.down + }, + { + id: '037', + name: 'add_trace_discovery', + up: migration037.up, + down: migration037.down + }, + { + id: '038', + name: 'add_external_node_enrichment', + up: migration038.up, + down: migration038.down + }, + { + id: '039', + name: 'add_percentile_latency', + up: migration039.up, + down: migration039.down + }, + { + id: '040', + name: 'add_span_storage', + up: migration040.up, + down: migration040.down + }, + { + id: '041', + name: 'add_span_retention_setting', + up: migration041.up, + down: migration041.down } ]; diff --git a/server/src/db/migrations/034_add_otel_sources.ts b/server/src/db/migrations/034_add_otel_sources.ts new file mode 100644 index 0000000..0d8297f --- /dev/null +++ b/server/src/db/migrations/034_add_otel_sources.ts @@ -0,0 +1,45 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + // DPS-77a: Add health_endpoint_format column to services + db.exec(` + ALTER TABLE services ADD COLUMN health_endpoint_format TEXT NOT NULL DEFAULT 'default' + `); + + // Backfill: services with schema_config should be 'schema' + db.exec(` + UPDATE services SET health_endpoint_format = 'schema' WHERE schema_config IS NOT NULL + `); + + // DPS-77b: Create team_api_keys table for OTLP push authentication + db.exec(` + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (created_by) REFERENCES users(id) + ) + `); + + // Unique index on key_hash for fast lookup during authentication + db.exec(`CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash)`); + + // Index for listing keys by team + db.exec(`CREATE INDEX idx_team_api_keys_team_id ON team_api_keys(team_id)`); +} + +export function down(db: Database): void { + // Drop team_api_keys table and indexes + db.exec(`DROP INDEX IF EXISTS idx_team_api_keys_team_id`); + db.exec(`DROP INDEX IF EXISTS idx_team_api_keys_key_hash`); + db.exec(`DROP TABLE IF EXISTS team_api_keys`); + + // Remove health_endpoint_format column from services + db.exec(`ALTER TABLE services DROP COLUMN health_endpoint_format`); +} diff --git a/server/src/db/migrations/035_api_key_rate_limit_columns.ts b/server/src/db/migrations/035_api_key_rate_limit_columns.ts new file mode 100644 index 0000000..f064f31 --- /dev/null +++ b/server/src/db/migrations/035_api_key_rate_limit_columns.ts @@ -0,0 +1,14 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + // DPS-84a: Add rate_limit_rpm column — NULL = system default, 0 = unlimited (admin-only), N = custom rpm + db.exec(`ALTER TABLE team_api_keys ADD COLUMN rate_limit_rpm INTEGER`); + + // DPS-84a: Add rate_limit_admin_locked column — 0 = unlocked, 1 = admin has locked against team edits + db.exec(`ALTER TABLE team_api_keys ADD COLUMN rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0`); +} + +export function down(db: Database): void { + db.exec(`ALTER TABLE team_api_keys DROP COLUMN rate_limit_admin_locked`); + db.exec(`ALTER TABLE team_api_keys DROP COLUMN rate_limit_rpm`); +} diff --git a/server/src/db/migrations/036_api_key_usage_buckets.ts b/server/src/db/migrations/036_api_key_usage_buckets.ts new file mode 100644 index 0000000..9ecc4a3 --- /dev/null +++ b/server/src/db/migrations/036_api_key_usage_buckets.ts @@ -0,0 +1,29 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + // DPS-85a: Create api_key_usage_buckets table for per-key usage tracking + // No ON DELETE CASCADE by design — orphaned rows for deleted keys are retained 7 days + // then pruned by the retention job + db.exec(` + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ) + `); + + // Composite index for per-key time-range queries + db.exec(`CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start)`); + + // Index on bucket_start alone to support retention DELETE statements + db.exec(`CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start)`); +} + +export function down(db: Database): void { + db.exec(`DROP INDEX IF EXISTS idx_usage_buckets_start`); + db.exec(`DROP INDEX IF EXISTS idx_usage_buckets_key_start`); + db.exec(`DROP TABLE IF EXISTS api_key_usage_buckets`); +} diff --git a/server/src/db/migrations/037_add_trace_discovery.ts b/server/src/db/migrations/037_add_trace_discovery.ts new file mode 100644 index 0000000..6af3d4f --- /dev/null +++ b/server/src/db/migrations/037_add_trace_discovery.ts @@ -0,0 +1,55 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + // Add discovery_source to dependencies + db.exec(`ALTER TABLE dependencies ADD COLUMN discovery_source TEXT NOT NULL DEFAULT 'manual'`); + + // Backfill: existing OTLP-pushed dependencies get 'otlp_metric' + db.exec(` + UPDATE dependencies SET discovery_source = 'otlp_metric' + WHERE service_id IN (SELECT id FROM services WHERE health_endpoint_format = 'otlp') + `); + + // User enrichment columns (separate from auto-detected values so trace pushes don't overwrite) + db.exec(`ALTER TABLE dependencies ADD COLUMN user_display_name TEXT`); + db.exec(`ALTER TABLE dependencies ADD COLUMN user_description TEXT`); + db.exec(`ALTER TABLE dependencies ADD COLUMN user_impact TEXT`); + + // Re-add auto-suggestion columns to dependency_associations (removed in 026) + db.exec(`ALTER TABLE dependency_associations ADD COLUMN is_auto_suggested INTEGER NOT NULL DEFAULT 0`); + db.exec(`ALTER TABLE dependency_associations ADD COLUMN is_dismissed INTEGER NOT NULL DEFAULT 0`); + + // Index for querying auto-suggested associations + db.exec(`CREATE INDEX idx_dep_assoc_auto_suggested ON dependency_associations(is_auto_suggested, is_dismissed)`); +} + +export function down(db: Database): void { + db.exec(`DROP INDEX IF EXISTS idx_dep_assoc_auto_suggested`); + + // SQLite doesn't support DROP COLUMN in all versions, so recreate dependency_associations + db.exec(` + CREATE TABLE dependency_associations_backup ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + linked_service_id TEXT NOT NULL, + association_type TEXT NOT NULL DEFAULT 'other', + manifest_managed INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE, + FOREIGN KEY (linked_service_id) REFERENCES services(id) ON DELETE CASCADE + ) + `); + db.exec(` + INSERT INTO dependency_associations_backup (id, dependency_id, linked_service_id, association_type, manifest_managed, created_at) + SELECT id, dependency_id, linked_service_id, association_type, manifest_managed, created_at + FROM dependency_associations + `); + db.exec(`DROP TABLE dependency_associations`); + db.exec(`ALTER TABLE dependency_associations_backup RENAME TO dependency_associations`); + db.exec(`CREATE INDEX idx_dep_associations_dependency_id ON dependency_associations(dependency_id)`); + db.exec(`CREATE INDEX idx_dep_associations_linked_service_id ON dependency_associations(linked_service_id)`); + + // SQLite doesn't support DROP COLUMN in all versions, so recreate dependencies without new columns + // For down migration simplicity, we just note these columns can't be easily removed in older SQLite + // In practice, rollback is rarely needed — the columns are nullable/defaulted and harmless +} diff --git a/server/src/db/migrations/038_add_external_node_enrichment.ts b/server/src/db/migrations/038_add_external_node_enrichment.ts new file mode 100644 index 0000000..a928f4c --- /dev/null +++ b/server/src/db/migrations/038_add_external_node_enrichment.ts @@ -0,0 +1,23 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + db.exec(` + CREATE TABLE external_node_enrichment ( + id TEXT PRIMARY KEY, + canonical_name TEXT NOT NULL UNIQUE, + display_name TEXT, + description TEXT, + impact TEXT, + contact TEXT, + service_type TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT, + FOREIGN KEY (updated_by) REFERENCES users(id) + ) + `); +} + +export function down(db: Database): void { + db.exec(`DROP TABLE IF EXISTS external_node_enrichment`); +} diff --git a/server/src/db/migrations/039_add_percentile_latency.ts b/server/src/db/migrations/039_add_percentile_latency.ts new file mode 100644 index 0000000..c2a7e9e --- /dev/null +++ b/server/src/db/migrations/039_add_percentile_latency.ts @@ -0,0 +1,17 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN p50_ms REAL`); + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN p95_ms REAL`); + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN p99_ms REAL`); + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN min_ms REAL`); + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN max_ms REAL`); + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN request_count INTEGER`); + db.exec(`ALTER TABLE dependency_latency_history ADD COLUMN source TEXT NOT NULL DEFAULT 'poll'`); +} + +export function down(_db: Database): void { + // SQLite doesn't support DROP COLUMN in all versions + // These columns are nullable/defaulted and harmless if left in place + // For a full rollback, recreate the table without these columns +} diff --git a/server/src/db/migrations/040_add_span_storage.ts b/server/src/db/migrations/040_add_span_storage.ts new file mode 100644 index 0000000..8dc1ef3 --- /dev/null +++ b/server/src/db/migrations/040_add_span_storage.ts @@ -0,0 +1,40 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + db.exec(` + CREATE TABLE spans ( + id TEXT PRIMARY KEY, + trace_id TEXT NOT NULL, + span_id TEXT NOT NULL, + parent_span_id TEXT, + service_name TEXT NOT NULL, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + kind INTEGER NOT NULL DEFAULT 0, + start_time TEXT NOT NULL, + end_time TEXT NOT NULL, + duration_ms REAL NOT NULL, + status_code INTEGER DEFAULT 0, + status_message TEXT, + attributes TEXT, + resource_attributes TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ) + `); + + db.exec(`CREATE INDEX idx_spans_trace_id ON spans(trace_id)`); + db.exec(`CREATE INDEX idx_spans_service_team ON spans(service_name, team_id)`); + db.exec(`CREATE INDEX idx_spans_start_time ON spans(start_time)`); + db.exec(`CREATE INDEX idx_spans_kind ON spans(kind)`); + db.exec(`CREATE INDEX idx_spans_created_at ON spans(created_at)`); +} + +export function down(db: Database): void { + db.exec(`DROP INDEX IF EXISTS idx_spans_created_at`); + db.exec(`DROP INDEX IF EXISTS idx_spans_kind`); + db.exec(`DROP INDEX IF EXISTS idx_spans_start_time`); + db.exec(`DROP INDEX IF EXISTS idx_spans_service_team`); + db.exec(`DROP INDEX IF EXISTS idx_spans_trace_id`); + db.exec(`DROP TABLE IF EXISTS spans`); +} diff --git a/server/src/db/migrations/041_add_span_retention_setting.ts b/server/src/db/migrations/041_add_span_retention_setting.ts new file mode 100644 index 0000000..cadec80 --- /dev/null +++ b/server/src/db/migrations/041_add_span_retention_setting.ts @@ -0,0 +1,20 @@ +import { Database } from 'better-sqlite3'; + +export function up(db: Database): void { + db.exec(` + CREATE TABLE IF NOT EXISTS app_settings ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL, + updated_at TEXT DEFAULT (datetime('now')), + updated_by TEXT, + FOREIGN KEY (updated_by) REFERENCES users(id) + ) + `); + + // Seed default span retention of 7 days + db.exec(`INSERT OR IGNORE INTO app_settings (key, value) VALUES ('span_retention_days', '7')`); +} + +export function down(db: Database): void { + db.exec(`DROP TABLE IF EXISTS app_settings`); +} diff --git a/server/src/db/migrations/__tests__/034_add_otel_sources.test.ts b/server/src/db/migrations/__tests__/034_add_otel_sources.test.ts new file mode 100644 index 0000000..16155f8 --- /dev/null +++ b/server/src/db/migrations/__tests__/034_add_otel_sources.test.ts @@ -0,0 +1,222 @@ +import Database from 'better-sqlite3'; +import { runMigrations } from '../../migrate'; +import { down } from '../034_add_otel_sources'; + +describe('034_add_otel_sources migration', () => { + let db: Database.Database; + + beforeEach(() => { + db = new Database(':memory:'); + db.pragma('foreign_keys = ON'); + // Run all migrations up to and including 034 + runMigrations(db); + }); + + afterEach(() => { + db.close(); + }); + + describe('up', () => { + it('should add health_endpoint_format column to services', () => { + const columns = db + .prepare("PRAGMA table_info('services')") + .all() as { name: string; type: string; notnull: number; dflt_value: string | null }[]; + + const formatCol = columns.find(c => c.name === 'health_endpoint_format'); + expect(formatCol).toBeDefined(); + expect(formatCol!.type).toBe('TEXT'); + expect(formatCol!.notnull).toBe(1); + expect(formatCol!.dflt_value).toBe("'default'"); + }); + + it('should create team_api_keys table', () => { + const tables = db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'team_api_keys'") + .all() as { name: string }[]; + + expect(tables).toHaveLength(1); + }); + + it('should create team_api_keys with correct columns', () => { + const columns = db + .prepare("PRAGMA table_info('team_api_keys')") + .all() as { name: string; type: string; notnull: number }[]; + + const colNames = columns.map(c => c.name); + expect(colNames).toContain('id'); + expect(colNames).toContain('team_id'); + expect(colNames).toContain('name'); + expect(colNames).toContain('key_hash'); + expect(colNames).toContain('key_prefix'); + expect(colNames).toContain('last_used_at'); + expect(colNames).toContain('created_at'); + expect(colNames).toContain('created_by'); + }); + + it('should create unique index on key_hash', () => { + const indexes = db + .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='team_api_keys'") + .all() as { name: string }[]; + + const hashIndex = indexes.find(i => i.name === 'idx_team_api_keys_key_hash'); + expect(hashIndex).toBeDefined(); + }); + + it('should create index on team_id', () => { + const indexes = db + .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='team_api_keys'") + .all() as { name: string }[]; + + const teamIndex = indexes.find(i => i.name === 'idx_team_api_keys_team_id'); + expect(teamIndex).toBeDefined(); + }); + + it('should default health_endpoint_format to default for new services', () => { + // Insert prerequisite team + db.prepare("INSERT INTO teams (id, name) VALUES ('team-1', 'Test Team')").run(); + + db.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, poll_interval_ms) + VALUES ('svc-1', 'Test Service', 'team-1', 'http://localhost/health', 30000) + `).run(); + + const service = db.prepare('SELECT health_endpoint_format FROM services WHERE id = ?').get('svc-1') as { + health_endpoint_format: string; + }; + expect(service.health_endpoint_format).toBe('default'); + }); + + it('should enforce foreign key on team_api_keys.team_id', () => { + expect(() => { + db.prepare(` + INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix) + VALUES ('key-1', 'non-existent', 'My Key', 'hash123', 'dps_a1b2') + `).run(); + }).toThrow(/FOREIGN KEY constraint failed/); + }); + + it('should enforce unique constraint on key_hash', () => { + db.prepare("INSERT INTO teams (id, name) VALUES ('team-1', 'Test Team')").run(); + db.prepare("INSERT INTO users (id, email, name, role) VALUES ('user-1', 'u@test.com', 'User', 'admin')").run(); + + db.prepare(` + INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_by) + VALUES ('key-1', 'team-1', 'Key 1', 'unique-hash', 'dps_a1b2', 'user-1') + `).run(); + + expect(() => { + db.prepare(` + INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_by) + VALUES ('key-2', 'team-1', 'Key 2', 'unique-hash', 'dps_c3d4', 'user-1') + `).run(); + }).toThrow(/UNIQUE constraint failed/); + }); + + it('should cascade delete team_api_keys when team is deleted', () => { + db.prepare("INSERT INTO teams (id, name) VALUES ('team-1', 'Test Team')").run(); + + db.prepare(` + INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix) + VALUES ('key-1', 'team-1', 'Key 1', 'hash-1', 'dps_a1b2') + `).run(); + + // Delete the team — should cascade + db.prepare("DELETE FROM teams WHERE id = 'team-1'").run(); + + const keys = db.prepare("SELECT * FROM team_api_keys WHERE team_id = 'team-1'").all(); + expect(keys).toHaveLength(0); + }); + }); + + describe('backfill', () => { + it('should set health_endpoint_format to schema for services with schema_config', () => { + db.prepare("INSERT INTO teams (id, name) VALUES ('team-1', 'Test Team')").run(); + + // Insert service with schema_config before migration + // Since migration already ran, we test by inserting with schema_config and checking + // We need to simulate pre-migration state — instead test by verifying the column behavior + db.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, poll_interval_ms, schema_config, health_endpoint_format) + VALUES ('svc-schema', 'Schema Service', 'team-1', 'http://localhost/health', 30000, '{"root":"$.deps"}', 'schema') + `).run(); + + db.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, poll_interval_ms, health_endpoint_format) + VALUES ('svc-default', 'Default Service', 'team-1', 'http://localhost/health', 30000, 'default') + `).run(); + + const schemaService = db.prepare('SELECT health_endpoint_format FROM services WHERE id = ?').get('svc-schema') as { + health_endpoint_format: string; + }; + expect(schemaService.health_endpoint_format).toBe('schema'); + + const defaultService = db.prepare('SELECT health_endpoint_format FROM services WHERE id = ?').get('svc-default') as { + health_endpoint_format: string; + }; + expect(defaultService.health_endpoint_format).toBe('default'); + }); + }); + + describe('backfill on pre-existing data', () => { + it('should correctly backfill when services exist before migration', () => { + // To test backfill properly, create a fresh DB, run migrations up to 033, + // insert data, then run migration 034 + const freshDb = new Database(':memory:'); + freshDb.pragma('foreign_keys = ON'); + + // Run all migrations — they include 034 which adds the column + runMigrations(freshDb); + + // We can verify the backfill logic by rolling back 034, inserting data, then re-running + // But since rollback + re-run is complex, let's verify the backfill SQL logic directly + freshDb.prepare("INSERT INTO teams (id, name) VALUES ('team-bf', 'Backfill Team')").run(); + + // Insert services — one with schema_config, one without + freshDb.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, poll_interval_ms, schema_config) + VALUES ('svc-with-schema', 'With Schema', 'team-bf', 'http://localhost/health', 30000, '{"root":"$.data"}') + `).run(); + + freshDb.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, poll_interval_ms) + VALUES ('svc-no-schema', 'No Schema', 'team-bf', 'http://localhost/health', 30000) + `).run(); + + // Re-run the backfill SQL to test its idempotency + freshDb.exec(` + UPDATE services SET health_endpoint_format = 'schema' WHERE schema_config IS NOT NULL + `); + + const withSchema = freshDb.prepare('SELECT health_endpoint_format FROM services WHERE id = ?').get('svc-with-schema') as { + health_endpoint_format: string; + }; + expect(withSchema.health_endpoint_format).toBe('schema'); + + const noSchema = freshDb.prepare('SELECT health_endpoint_format FROM services WHERE id = ?').get('svc-no-schema') as { + health_endpoint_format: string; + }; + expect(noSchema.health_endpoint_format).toBe('default'); + + freshDb.close(); + }); + }); + + describe('down', () => { + it('should remove team_api_keys table and health_endpoint_format column', () => { + down(db); + + // team_api_keys should not exist + const tables = db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'team_api_keys'") + .all(); + expect(tables).toHaveLength(0); + + // health_endpoint_format column should not exist + const columns = db + .prepare("PRAGMA table_info('services')") + .all() as { name: string }[]; + const formatCol = columns.find(c => c.name === 'health_endpoint_format'); + expect(formatCol).toBeUndefined(); + }); + }); +}); diff --git a/server/src/db/migrations/__tests__/037_041_trace_discovery.test.ts b/server/src/db/migrations/__tests__/037_041_trace_discovery.test.ts new file mode 100644 index 0000000..9275959 --- /dev/null +++ b/server/src/db/migrations/__tests__/037_041_trace_discovery.test.ts @@ -0,0 +1,249 @@ +import Database from 'better-sqlite3'; +import { runMigrations } from '../../migrate'; + +describe('migrations 037-041 (trace discovery)', () => { + let db: Database.Database; + + beforeEach(() => { + db = new Database(':memory:'); + db.pragma('foreign_keys = ON'); + runMigrations(db); + }); + + afterEach(() => { + db.close(); + }); + + describe('037: add_trace_discovery', () => { + it('adds discovery_source column to dependencies with correct default', () => { + const columns = db + .prepare("PRAGMA table_info('dependencies')") + .all() as { name: string; type: string; notnull: number; dflt_value: string | null }[]; + + const col = columns.find(c => c.name === 'discovery_source'); + expect(col).toBeDefined(); + expect(col!.type).toBe('TEXT'); + expect(col!.notnull).toBe(1); + expect(col!.dflt_value).toBe("'manual'"); + }); + + it('adds user enrichment columns to dependencies', () => { + const columns = db + .prepare("PRAGMA table_info('dependencies')") + .all() as { name: string }[]; + const colNames = columns.map(c => c.name); + + expect(colNames).toContain('user_display_name'); + expect(colNames).toContain('user_description'); + expect(colNames).toContain('user_impact'); + }); + + it('adds is_auto_suggested and is_dismissed to dependency_associations', () => { + const columns = db + .prepare("PRAGMA table_info('dependency_associations')") + .all() as { name: string; type: string; notnull: number; dflt_value: string | null }[]; + + const autoSuggested = columns.find(c => c.name === 'is_auto_suggested'); + expect(autoSuggested).toBeDefined(); + expect(autoSuggested!.notnull).toBe(1); + expect(autoSuggested!.dflt_value).toBe('0'); + + const dismissed = columns.find(c => c.name === 'is_dismissed'); + expect(dismissed).toBeDefined(); + expect(dismissed!.notnull).toBe(1); + expect(dismissed!.dflt_value).toBe('0'); + }); + + it('creates idx_dep_assoc_auto_suggested index', () => { + const indexes = db + .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='dependency_associations'") + .all() as { name: string }[]; + + expect(indexes.find(i => i.name === 'idx_dep_assoc_auto_suggested')).toBeDefined(); + }); + + it('backfills discovery_source to otlp_metric for OTLP services', () => { + db.prepare("INSERT INTO teams (id, name) VALUES ('t1', 'Team')").run(); + db.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, poll_interval_ms, health_endpoint_format) + VALUES ('svc-otlp', 'OTLP Service', 't1', 'otlp-push', 30000, 'otlp') + `).run(); + db.prepare(` + INSERT INTO dependencies (id, service_id, name) VALUES ('dep-1', 'svc-otlp', 'postgres') + `).run(); + + // Re-run the backfill to verify logic + db.exec(` + UPDATE dependencies SET discovery_source = 'otlp_metric' + WHERE service_id IN (SELECT id FROM services WHERE health_endpoint_format = 'otlp') + `); + + const dep = db.prepare('SELECT discovery_source FROM dependencies WHERE id = ?').get('dep-1') as { discovery_source: string }; + expect(dep.discovery_source).toBe('otlp_metric'); + }); + }); + + describe('038: add_external_node_enrichment', () => { + it('creates external_node_enrichment table', () => { + const tables = db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'external_node_enrichment'") + .all() as { name: string }[]; + expect(tables).toHaveLength(1); + }); + + it('has correct columns', () => { + const columns = db + .prepare("PRAGMA table_info('external_node_enrichment')") + .all() as { name: string }[]; + const colNames = columns.map(c => c.name); + + expect(colNames).toContain('id'); + expect(colNames).toContain('canonical_name'); + expect(colNames).toContain('display_name'); + expect(colNames).toContain('description'); + expect(colNames).toContain('impact'); + expect(colNames).toContain('contact'); + expect(colNames).toContain('service_type'); + expect(colNames).toContain('updated_by'); + }); + + it('enforces unique canonical_name', () => { + db.prepare(` + INSERT INTO external_node_enrichment (id, canonical_name) VALUES ('e1', 'PostgreSQL') + `).run(); + + expect(() => { + db.prepare(` + INSERT INTO external_node_enrichment (id, canonical_name) VALUES ('e2', 'PostgreSQL') + `).run(); + }).toThrow(/UNIQUE constraint failed/); + }); + }); + + describe('039: add_percentile_latency', () => { + it('adds percentile columns to dependency_latency_history', () => { + const columns = db + .prepare("PRAGMA table_info('dependency_latency_history')") + .all() as { name: string; type: string; dflt_value: string | null }[]; + const colNames = columns.map(c => c.name); + + expect(colNames).toContain('p50_ms'); + expect(colNames).toContain('p95_ms'); + expect(colNames).toContain('p99_ms'); + expect(colNames).toContain('min_ms'); + expect(colNames).toContain('max_ms'); + expect(colNames).toContain('request_count'); + }); + + it('adds source column with poll default', () => { + const columns = db + .prepare("PRAGMA table_info('dependency_latency_history')") + .all() as { name: string; notnull: number; dflt_value: string | null }[]; + + const sourceCol = columns.find(c => c.name === 'source'); + expect(sourceCol).toBeDefined(); + expect(sourceCol!.notnull).toBe(1); + expect(sourceCol!.dflt_value).toBe("'poll'"); + }); + + it('percentile columns are nullable', () => { + const columns = db + .prepare("PRAGMA table_info('dependency_latency_history')") + .all() as { name: string; notnull: number }[]; + + for (const name of ['p50_ms', 'p95_ms', 'p99_ms', 'min_ms', 'max_ms', 'request_count']) { + const col = columns.find(c => c.name === name); + expect(col).toBeDefined(); + expect(col!.notnull).toBe(0); + } + }); + }); + + describe('040: add_span_storage', () => { + it('creates spans table', () => { + const tables = db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'spans'") + .all() as { name: string }[]; + expect(tables).toHaveLength(1); + }); + + it('has correct columns', () => { + const columns = db + .prepare("PRAGMA table_info('spans')") + .all() as { name: string }[]; + const colNames = columns.map(c => c.name); + + expect(colNames).toContain('id'); + expect(colNames).toContain('trace_id'); + expect(colNames).toContain('span_id'); + expect(colNames).toContain('parent_span_id'); + expect(colNames).toContain('service_name'); + expect(colNames).toContain('team_id'); + expect(colNames).toContain('name'); + expect(colNames).toContain('kind'); + expect(colNames).toContain('start_time'); + expect(colNames).toContain('end_time'); + expect(colNames).toContain('duration_ms'); + expect(colNames).toContain('status_code'); + expect(colNames).toContain('status_message'); + expect(colNames).toContain('attributes'); + expect(colNames).toContain('resource_attributes'); + }); + + it('creates all expected indexes', () => { + const indexes = db + .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='spans'") + .all() as { name: string }[]; + const indexNames = indexes.map(i => i.name); + + expect(indexNames).toContain('idx_spans_trace_id'); + expect(indexNames).toContain('idx_spans_service_team'); + expect(indexNames).toContain('idx_spans_start_time'); + expect(indexNames).toContain('idx_spans_kind'); + expect(indexNames).toContain('idx_spans_created_at'); + }); + + it('cascades delete on team removal', () => { + db.prepare("INSERT INTO teams (id, name) VALUES ('t1', 'Team')").run(); + db.prepare(` + INSERT INTO spans (id, trace_id, span_id, service_name, team_id, name, kind, start_time, end_time, duration_ms) + VALUES ('s1', 'trace-1', 'span-1', 'svc', 't1', 'GET /api', 2, '2024-01-01T00:00:00Z', '2024-01-01T00:00:01Z', 1000) + `).run(); + + db.prepare("DELETE FROM teams WHERE id = 't1'").run(); + + const spans = db.prepare('SELECT * FROM spans').all(); + expect(spans).toHaveLength(0); + }); + }); + + describe('041: add_span_retention_setting', () => { + it('creates app_settings table', () => { + const tables = db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'app_settings'") + .all() as { name: string }[]; + expect(tables).toHaveLength(1); + }); + + it('seeds span_retention_days with default value of 7', () => { + const row = db + .prepare("SELECT value FROM app_settings WHERE key = 'span_retention_days'") + .get() as { value: string }; + + expect(row).toBeDefined(); + expect(row.value).toBe('7'); + }); + + it('has correct columns', () => { + const columns = db + .prepare("PRAGMA table_info('app_settings')") + .all() as { name: string }[]; + const colNames = columns.map(c => c.name); + + expect(colNames).toContain('key'); + expect(colNames).toContain('value'); + expect(colNames).toContain('updated_at'); + expect(colNames).toContain('updated_by'); + }); + }); +}); diff --git a/server/src/db/types.ts b/server/src/db/types.ts index 3d403a7..db8155f 100644 --- a/server/src/db/types.ts +++ b/server/src/db/types.ts @@ -91,6 +91,20 @@ export interface SchemaMapping { }; } +// Metric schema config for Prometheus and OTLP custom metric/label mappings +export interface MetricSchemaConfig { + metrics: Record; // user metric name → depsera field (state, healthy, latency, code, skipped) + labels: Record; // user label/attribute name → depsera field (name, type, impact, description, errorMessage) + latency_unit?: 'ms' | 's'; // default 'ms' + healthy_value?: number; // value that means healthy for the 'healthy' target (default: 1) +} + +export const VALID_METRIC_TARGETS = ['state', 'healthy', 'latency', 'code', 'skipped'] as const; +export const VALID_LABEL_TARGETS = ['name', 'type', 'impact', 'description', 'errorMessage'] as const; + +// Health endpoint format types +export type HealthEndpointFormat = 'default' | 'schema' | 'prometheus' | 'otlp'; + // Service types export interface Service { id: string; @@ -108,6 +122,7 @@ export interface Service { poll_warnings: string | null; // JSON array of warning strings manifest_key: string | null; manifest_managed: number; // SQLite boolean — 1 if managed by manifest + health_endpoint_format: HealthEndpointFormat; manifest_config_id: string | null; // FK → team_manifest_config.id manifest_last_synced_values: string | null; // JSON snapshot of last synced field values created_at: string; @@ -138,6 +153,9 @@ export interface ServiceWithDependencies extends Service { team: Team; } +// Discovery source types +export type DiscoverySource = 'manual' | 'otlp_metric' | 'otlp_trace'; + // Dependency types export type HealthState = 0 | 1 | 2; // 0=OK, 1=WARNING, 2=CRITICAL @@ -174,6 +192,10 @@ export interface Dependency { check_details: string | null; // JSON string of check details error: string | null; // JSON string of error object error_message: string | null; + discovery_source: DiscoverySource; + user_display_name: string | null; + user_description: string | null; + user_impact: string | null; skipped: number; // SQLite boolean — 1 if health check is skipped last_checked: string | null; last_status_change: string | null; @@ -208,6 +230,8 @@ export interface DependencyAssociation { dependency_id: string; linked_service_id: string; association_type: AssociationType; + is_auto_suggested: number; // SQLite boolean — 1 if auto-suggested from traces + is_dismissed: number; // SQLite boolean — 1 if user dismissed the suggestion manifest_managed: number; // SQLite boolean — 1 if managed by manifest created_at: string; } @@ -256,6 +280,14 @@ export interface ProactiveDepsStatus { code: number; latency: number; skipped?: boolean; + percentiles?: { + p50?: number; + p95?: number; + p99?: number; + min?: number; + max?: number; + requestCount?: number; + }; }; lastChecked: string; checkDetails?: Record; @@ -313,6 +345,67 @@ export interface LatencyDataPoint { recorded_at: string; } +// Span types (full span storage for trace correlation) +export interface Span { + id: string; + trace_id: string; + span_id: string; + parent_span_id: string | null; + service_name: string; + team_id: string; + name: string; + kind: number; // 0=UNSPECIFIED, 1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER + start_time: string; + end_time: string; + duration_ms: number; + status_code: number; + status_message: string | null; + attributes: string | null; // JSON + resource_attributes: string | null; // JSON + created_at: string; +} + +export interface CreateSpanInput { + trace_id: string; + span_id: string; + parent_span_id?: string | null; + service_name: string; + team_id: string; + name: string; + kind?: number; + start_time: string; + end_time: string; + duration_ms: number; + status_code?: number; + status_message?: string | null; + attributes?: string | null; + resource_attributes?: string | null; +} + +// External node enrichment types (org-wide enrichment for virtual external nodes) +export interface ExternalNodeEnrichment { + id: string; + canonical_name: string; + display_name: string | null; + description: string | null; + impact: string | null; + contact: string | null; // JSON + service_type: string | null; + created_at: string; + updated_at: string; + updated_by: string | null; +} + +export interface UpsertExternalNodeEnrichmentInput { + canonical_name: string; + display_name?: string | null; + description?: string | null; + impact?: string | null; + contact?: string | null; + service_type?: string | null; + updated_by?: string | null; +} + // Error history types export interface DependencyErrorHistory { id: string; @@ -369,9 +462,11 @@ export type AuditAction = | 'drift.bulk_accepted' | 'drift.bulk_dismissed' | 'alert_mute.created' - | 'alert_mute.deleted'; + | 'alert_mute.deleted' + | 'api_key.created' + | 'api_key.revoked'; -export type AuditResourceType = 'user' | 'team' | 'service' | 'external_service' | 'settings' | 'canonical_override' | 'dependency' | 'manifest_config' | 'drift_flag' | 'alert_mute'; +export type AuditResourceType = 'user' | 'team' | 'service' | 'external_service' | 'settings' | 'canonical_override' | 'dependency' | 'manifest_config' | 'drift_flag' | 'alert_mute' | 'team_api_key'; export interface AuditLogEntry { id: string; @@ -477,6 +572,34 @@ export interface CreateAlertMuteInput { expires_at?: string | null; } +// Team API key types +export interface TeamApiKey { + id: string; + team_id: string; + name: string; + key_hash: string; + key_prefix: string; + rate_limit_rpm: number | null; // null = system default + rate_limit_admin_locked: number; // 0 or 1 (SQLite boolean) + last_used_at: string | null; + created_at: string; + created_by: string | null; +} + +export interface CreateTeamApiKeyInput { + team_id: string; + name: string; + created_by?: string; +} + +export interface ApiKeyUsageBucket { + api_key_id: string; + bucket_start: string; // ISO 8601 UTC, e.g. "2025-01-15T14:32:00" + granularity: 'minute' | 'hour'; + push_count: number; + rejected_count: number; +} + // Status change event types export interface StatusChangeEventRow { id: string; diff --git a/server/src/index.ts b/server/src/index.ts index e8b042e..e9ce60d 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -26,6 +26,7 @@ import activityRouter from './routes/activity'; import { manifestTeamRouter, manifestRouter } from './routes/manifest'; import { driftRouter } from './routes/drifts'; import catalogRouter from './routes/catalog'; +import externalNodesRouter from './routes/externalNodes'; import { HealthPollingService, PollingEventType, StatusChangeEvent, PollCompleteEvent } from './services/polling'; import { getServicePollHistoryRecorder } from './services/polling/ServicePollHistoryRecorder'; import { SettingsService } from './services/settings/SettingsService'; @@ -41,7 +42,12 @@ import { csrfProtection } from './middleware/csrf'; import { createSecurityHeaders } from './middleware/securityHeaders'; import { parseTrustProxy } from './middleware/trustProxy'; import { createHttpsRedirect } from './middleware/httpsRedirect'; -import { createGlobalRateLimit, createAuthRateLimit } from './middleware/rateLimit'; +import { createGlobalRateLimit, createAuthRateLimit, createOtlpGlobalRateLimit } from './middleware/rateLimit'; +import { createPerKeyRateLimit } from './middleware/perKeyRateLimit'; +import { createTrackApiKeyUsage, startUsageFlusher } from './middleware/trackApiKeyUsage'; +import { requireApiKeyAuth } from './auth/apiKeyAuth'; +import otlpRouter from './routes/otlp'; +import traceRouter from './routes/otlp/traces'; import { createRequestLogger } from './middleware/requestLogger'; import logger from './utils/logger'; @@ -64,6 +70,26 @@ app.use(cors({ credentials: true, })); app.use(express.json({ limit: '100kb' })); + +// OTLP receiver — mounted before session/CSRF middleware (collectors use API key auth, not sessions) +app.use('/v1/metrics', + express.json({ limit: '1mb' }), + createOtlpGlobalRateLimit(), + requireApiKeyAuth, + createPerKeyRateLimit(), + createTrackApiKeyUsage(), + otlpRouter, +); + +app.use('/v1/traces', + express.json({ limit: '2mb' }), + createOtlpGlobalRateLimit(), + requireApiKeyAuth, + createPerKeyRateLimit(), + createTrackApiKeyUsage(), + traceRouter, +); + app.use(createGlobalRateLimit()); app.use(sessionMiddleware); @@ -97,6 +123,7 @@ app.use('/api/teams', requireAuth, manifestTeamRouter); app.use('/api/teams', requireAuth, driftRouter); app.use('/api/manifest', requireAuth, manifestRouter); app.use('/api/catalog', requireAuth, catalogRouter); +app.use('/api/external-nodes', requireAuth, externalNodesRouter); // Global error handler — catches body-parser errors, unhandled route errors, etc. // Must be registered after all routes (Express identifies error handlers by 4-param signature). @@ -192,6 +219,9 @@ async function start() { // Start polling all active services pollingService.startAll(); + // Start the OTLP API key usage flusher (persists in-memory counts to DB on an interval) + startUsageFlusher(); + let server: http.Server | https.Server; let httpRedirectServer: http.Server | undefined; diff --git a/server/src/middleware/perKeyRateLimit.test.ts b/server/src/middleware/perKeyRateLimit.test.ts new file mode 100644 index 0000000..a8ede85 --- /dev/null +++ b/server/src/middleware/perKeyRateLimit.test.ts @@ -0,0 +1,406 @@ +import express from 'express'; +import request from 'supertest'; +import { createPerKeyRateLimit, evictBucket } from './perKeyRateLimit'; + +// Mock the stores module +const mockFindById = jest.fn(); +jest.mock('../stores', () => ({ + getStores: () => ({ + teamApiKeys: { + findById: mockFindById, + }, + }), +})); + +// Mock trackApiKeyUsage to capture incrementRejected calls +const mockIncrementRejected = jest.fn(); +jest.mock('./trackApiKeyUsage', () => ({ + incrementRejected: mockIncrementRejected, +})); + +const BASE_KEY = { + id: 'key-1', + team_id: 'team-1', + name: 'Test Key', + key_hash: 'abc123', + key_prefix: 'dps_test', + rate_limit_rpm: null as number | null, + rate_limit_admin_locked: 0, + last_used_at: null, + created_at: '2025-01-01T00:00:00', + created_by: 'user-1', +}; + +function createApp(opts: { + getNow?: () => number; + nodeEnv?: string; + envRpm?: string; + burstSeconds?: string; + warningThreshold?: string; +} = {}) { + const originalEnv = { ...process.env }; + if (opts.nodeEnv !== undefined) process.env.NODE_ENV = opts.nodeEnv; + if (opts.envRpm !== undefined) process.env.OTLP_PER_KEY_RATE_LIMIT_RPM = opts.envRpm; + if (opts.burstSeconds !== undefined) process.env.OTLP_RATE_LIMIT_BURST_SECONDS = opts.burstSeconds; + if (opts.warningThreshold !== undefined) process.env.OTLP_RATE_LIMIT_WARNING_THRESHOLD = opts.warningThreshold; + + const app = express(); + app.use(express.json()); + + // Inject apiKeyId on every request + app.use((req, _res, next) => { + req.apiKeyId = 'key-1'; + next(); + }); + + app.use(createPerKeyRateLimit({ getNow: opts.getNow })); + + app.post('/v1/metrics', (_req, res) => { + res.json({ ok: true }); + }); + + // Cleanup function to restore env + const cleanup = () => { + process.env = originalEnv; + }; + + return { app, cleanup }; +} + +describe('perKeyRateLimit', () => { + const originalEnv = { ...process.env }; + + beforeEach(() => { + jest.clearAllMocks(); + mockFindById.mockReturnValue({ ...BASE_KEY }); + // Reset module-level state by evicting the key + evictBucket('key-1'); + // Ensure we're not in dev mode + process.env.NODE_ENV = 'test'; + // Clean rate limit env vars + delete process.env.OTLP_PER_KEY_RATE_LIMIT_RPM; + delete process.env.OTLP_RATE_LIMIT_BURST_SECONDS; + delete process.env.OTLP_RATE_LIMIT_WARNING_THRESHOLD; + }); + + afterAll(() => { + process.env = originalEnv; + }); + + describe('allow path and response headers (DPS-100a)', () => { + it('should allow requests within limit and call next()', async () => { + // Small burst so tests are fast: 60 rpm, 6s burst = capacity 6 + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.status).toBe(200); + expect(res.body.ok).toBe(true); + }); + + it('should set RateLimit-Limit header', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['ratelimit-limit']).toBe('60'); + }); + + it('should set RateLimit-Remaining header', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + // capacity = ceil(60/60 * 6) = 6, after consuming 1 token => remaining = 5 + expect(res.headers['ratelimit-remaining']).toBe('5'); + }); + + it('should set RateLimit-Reset header', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['ratelimit-reset']).toBeDefined(); + const resetValue = parseInt(res.headers['ratelimit-reset'] as string, 10); + expect(resetValue).toBeGreaterThan(0); + }); + + it('should set X-RateLimit-Key header with key prefix', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['x-ratelimit-key']).toBe('dps_test'); + }); + + it('should not set Retry-After header on allowed requests', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['retry-after']).toBeUndefined(); + }); + }); + + describe('429 rejection with OTLP body (DPS-100b)', () => { + it('should return 429 when bucket is exhausted', async () => { + // capacity = ceil(60/60 * 6) = 6 + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + // Exhaust all 6 tokens + for (let i = 0; i < 6; i++) { + await request(app).post('/v1/metrics').send({}); + } + + // 7th request should be rejected + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.status).toBe(429); + }); + + it('should return OTLP partialSuccess body on 429', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + // Exhaust all 6 tokens + for (let i = 0; i < 6; i++) { + await request(app).post('/v1/metrics').send({}); + } + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.body.partialSuccess).toBeDefined(); + expect(res.body.partialSuccess.rejectedDataPoints).toBe(0); + expect(res.body.partialSuccess.errorMessage).toMatch(/Rate limit exceeded/); + expect(res.body.partialSuccess.errorMessage).toContain('dps_test'); + }); + + it('should set Retry-After header on 429', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + for (let i = 0; i < 6; i++) { + await request(app).post('/v1/metrics').send({}); + } + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.status).toBe(429); + expect(res.headers['retry-after']).toBeDefined(); + const retryAfter = parseInt(res.headers['retry-after'] as string, 10); + expect(retryAfter).toBeGreaterThan(0); + }); + + it('should set RateLimit-Remaining to 0 on 429', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + for (let i = 0; i < 6; i++) { + await request(app).post('/v1/metrics').send({}); + } + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.status).toBe(429); + expect(res.headers['ratelimit-remaining']).toBe('0'); + }); + + it('should call incrementRejected on 429', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + for (let i = 0; i < 6; i++) { + await request(app).post('/v1/metrics').send({}); + } + + await request(app).post('/v1/metrics').send({}); + + expect(mockIncrementRejected).toHaveBeenCalledWith('key-1'); + }); + }); + + describe('rpm variants (DPS-100c)', () => { + it('should use custom rate_limit_rpm from key over system default', async () => { + // Custom 120 rpm => capacity = ceil(120/60 * 6) = 12 + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 120 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['ratelimit-limit']).toBe('120'); + }); + + it('should fall back to OTLP_PER_KEY_RATE_LIMIT_RPM env var when rate_limit_rpm is null', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: null }); + process.env.OTLP_PER_KEY_RATE_LIMIT_RPM = '300'; + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['ratelimit-limit']).toBe('300'); + }); + + it('should bypass limiter entirely when rate_limit_rpm is 0', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 0 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + // Send many requests — all should pass with no rate limit headers + for (let i = 0; i < 100; i++) { + const res = await request(app).post('/v1/metrics').send({}); + expect(res.status).toBe(200); + } + }); + }); + + describe('soft limit warning header (DPS-100d)', () => { + it('should set X-RateLimit-Warning when consumption exceeds 80%', async () => { + // capacity = ceil(60/60 * 6) = 6, 80% consumed = 4.8 tokens used + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + // Consume 5 tokens (5/6 = 83% consumed) + for (let i = 0; i < 4; i++) { + await request(app).post('/v1/metrics').send({}); + } + + // 5th request should trigger warning (consumed = 5/6 = 83%) + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['x-ratelimit-warning']).toBe('true'); + }); + + it('should not set X-RateLimit-Warning on fresh bucket with single request', async () => { + // capacity = 6, 1 consumed = 16.7% + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['x-ratelimit-warning']).toBeUndefined(); + }); + }); + + describe('burst capacity (DPS-100e)', () => { + it('should allow exactly capacity number of requests in a burst', async () => { + // 60 rpm, 6s burst => capacity = 6 + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + const { app } = createApp({ getNow: () => now }); + + // All 6 should pass + for (let i = 0; i < 6; i++) { + const res = await request(app).post('/v1/metrics').send({}); + expect(res.status).toBe(200); + } + + // 7th fails + const res = await request(app).post('/v1/metrics').send({}); + expect(res.status).toBe(429); + }); + + it('should refill tokens over time and allow new requests', async () => { + // 60 rpm => 1 req/sec, capacity = 6 + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + let now = 1000000; + const { app } = createApp({ getNow: () => now }); + + // Exhaust all 6 tokens + for (let i = 0; i < 6; i++) { + await request(app).post('/v1/metrics').send({}); + } + + // Advance 2 seconds => 2 tokens refilled + now += 2000; + const res = await request(app).post('/v1/metrics').send({}); + expect(res.status).toBe(200); + }); + }); + + describe('evictBucket re-reads from DB (DPS-100f)', () => { + it('should re-read rate limit from store after eviction', async () => { + // Start with 60 rpm + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + let now = 1000000; + const { app } = createApp({ getNow: () => now }); + + const res1 = await request(app).post('/v1/metrics').send({}); + expect(res1.headers['ratelimit-limit']).toBe('60'); + + // Simulate DB update: change to 120 rpm + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 120 }); + evictBucket('key-1'); + + // Next request should pick up new limit + now += 100; + const res2 = await request(app).post('/v1/metrics').send({}); + expect(res2.headers['ratelimit-limit']).toBe('120'); + }); + + it('should call findById again after eviction', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + let now = 1000000; + const { app } = createApp({ getNow: () => now }); + + await request(app).post('/v1/metrics').send({}); + const callCountBefore = mockFindById.mock.calls.length; + + evictBucket('key-1'); + now += 100; + await request(app).post('/v1/metrics').send({}); + + // Should have called findById at least once more after eviction + expect(mockFindById.mock.calls.length).toBeGreaterThan(callCountBefore); + }); + }); + + describe('dev mode bypass (DPS-100g)', () => { + it('should skip rate limiting in development mode', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + process.env.NODE_ENV = 'development'; + const { app } = createApp({ getNow: () => now, nodeEnv: 'development' }); + + // Send more requests than capacity — all should pass + for (let i = 0; i < 20; i++) { + const res = await request(app).post('/v1/metrics').send({}); + expect(res.status).toBe(200); + } + }); + + it('should not set rate limit headers in development mode', async () => { + mockFindById.mockReturnValue({ ...BASE_KEY, rate_limit_rpm: 60 }); + const now = 1000000; + process.env.NODE_ENV = 'development'; + const { app } = createApp({ getNow: () => now, nodeEnv: 'development' }); + + const res = await request(app).post('/v1/metrics').send({}); + + expect(res.headers['ratelimit-limit']).toBeUndefined(); + expect(res.headers['ratelimit-remaining']).toBeUndefined(); + expect(res.headers['x-ratelimit-key']).toBeUndefined(); + }); + }); +}); diff --git a/server/src/middleware/perKeyRateLimit.ts b/server/src/middleware/perKeyRateLimit.ts new file mode 100644 index 0000000..a29e2b6 --- /dev/null +++ b/server/src/middleware/perKeyRateLimit.ts @@ -0,0 +1,162 @@ +import { Request, Response, NextFunction, RequestHandler } from 'express'; +import { TeamApiKey } from '../db/types'; +import { getStores } from '../stores'; +import logger from '../utils/logger'; + +interface TokenBucket { + tokens: number; + capacity: number; + refillRatePerMs: number; + lastRefillAt: number; + effectiveRpm: number; +} + +const buckets = new Map(); +const lastAlertAt = new Map(); + +export function evictBucket(apiKeyId: string): void { + buckets.delete(apiKeyId); +} + +function getEffectiveRpm(key: TeamApiKey): number { + if (key.rate_limit_rpm !== null) return key.rate_limit_rpm; + return parseInt(process.env.OTLP_PER_KEY_RATE_LIMIT_RPM ?? '150000', 10); +} + +function getOrCreateBucket(apiKeyId: string, getNow: () => number): TokenBucket | null { + let bucket = buckets.get(apiKeyId); + if (bucket) return bucket; + + const stores = getStores(); + const key = stores.teamApiKeys.findById(apiKeyId); + if (!key) return null; + + const effectiveRpm = getEffectiveRpm(key); + + // rpm = 0 means unlimited (admin-only) — no bucket needed + if (effectiveRpm === 0) return null; + + const refillRatePerSec = effectiveRpm / 60; + const burstSeconds = parseInt(process.env.OTLP_RATE_LIMIT_BURST_SECONDS ?? '6', 10); + const capacity = Math.ceil(refillRatePerSec * burstSeconds); + + bucket = { + tokens: capacity, + capacity, + refillRatePerMs: refillRatePerSec / 1000, + lastRefillAt: getNow(), + effectiveRpm, + }; + + buckets.set(apiKeyId, bucket); + return bucket; +} + +function refillBucket(bucket: TokenBucket, now: number): void { + const elapsed = now - bucket.lastRefillAt; + bucket.tokens = Math.min(bucket.capacity, bucket.tokens + elapsed * bucket.refillRatePerMs); + bucket.lastRefillAt = now; +} + +function setRateLimitHeaders(res: Response, bucket: TokenBucket, keyPrefix: string, now: number): void { + res.setHeader('RateLimit-Limit', bucket.effectiveRpm); + res.setHeader('RateLimit-Remaining', Math.max(0, Math.floor(bucket.tokens))); + res.setHeader( + 'RateLimit-Reset', + Math.ceil(now / 1000 + (bucket.capacity - bucket.tokens) / bucket.refillRatePerMs / 1000), + ); + res.setHeader('X-RateLimit-Key', keyPrefix); +} + +export function createPerKeyRateLimit(options?: { getNow?: () => number }): RequestHandler { + const getNow = options?.getNow ?? Date.now; + + return (req: Request, res: Response, next: NextFunction): void => { + if (process.env.NODE_ENV === 'development') { + next(); + return; + } + + const apiKeyId = req.apiKeyId; + if (!apiKeyId) { + next(); + return; + } + + const stores = getStores(); + const key = stores.teamApiKeys.findById(apiKeyId); + if (!key) { + next(); + return; + } + + // rpm = 0 means unlimited — skip rate limiting entirely + if (key.rate_limit_rpm === 0) { + next(); + return; + } + + const bucket = getOrCreateBucket(apiKeyId, getNow); + if (!bucket) { + next(); + return; + } + + const now = getNow(); + refillBucket(bucket, now); + + const WARNING_THRESHOLD = parseFloat(process.env.OTLP_RATE_LIMIT_WARNING_THRESHOLD ?? '0.80'); + + if (bucket.tokens < 1) { + // Rejection path — 429 + const retryAfterSec = Math.ceil((1 - bucket.tokens) / bucket.refillRatePerMs / 1000); + res.setHeader('Retry-After', retryAfterSec); + setRateLimitHeaders(res, bucket, key.key_prefix, now); + + // Track rejected request in usage accumulator + try { + // Dynamic import to avoid circular dependency at module load time + // eslint-disable-next-line @typescript-eslint/no-var-requires + const { incrementRejected } = require('./trackApiKeyUsage'); + incrementRejected(apiKeyId); + } catch { + // trackApiKeyUsage not yet loaded — skip + } + + res.status(429).json({ + partialSuccess: { + rejectedDataPoints: 0, + errorMessage: `Rate limit exceeded for API key ${key.key_prefix}. Limit: ${bucket.effectiveRpm} req/min. Retry-After: ${retryAfterSec}s.`, + }, + }); + return; + } + + // Allow path — consume one token + bucket.tokens -= 1; + setRateLimitHeaders(res, bucket, key.key_prefix, now); + + // Check soft limit warning + const consumedRatio = (bucket.capacity - bucket.tokens) / bucket.capacity; + if (consumedRatio >= WARNING_THRESHOLD) { + res.setHeader('X-RateLimit-Warning', 'true'); + + // Alert debounce — fire at most once per 15 minutes per key + const ALERT_DEBOUNCE_MS = 15 * 60 * 1000; + const last = lastAlertAt.get(apiKeyId) ?? 0; + if (now - last > ALERT_DEBOUNCE_MS) { + lastAlertAt.set(apiKeyId, now); + const used = bucket.capacity - Math.floor(bucket.tokens); + const pct = Math.round(consumedRatio * 100); + setImmediate(() => { + logger.warn( + { apiKeyId, keyPrefix: key.key_prefix, consumedPct: pct, effectiveRpm: bucket.effectiveRpm }, + `API key \`${key.key_prefix}\` is at ${pct}% of its rate limit (${used}/${bucket.effectiveRpm} req/min).`, + ); + }); + } + } + + next(); + }; +} diff --git a/server/src/middleware/rateLimit.ts b/server/src/middleware/rateLimit.ts index 70933b7..196c86c 100644 --- a/server/src/middleware/rateLimit.ts +++ b/server/src/middleware/rateLimit.ts @@ -46,3 +46,17 @@ export function createAuthRateLimit(config?: Partial) { message: { error: 'Too many authentication attempts, please try again later' }, }); } + +export function createOtlpGlobalRateLimit(config?: Partial) { + const windowMs = config?.windowMs ?? parseInt(process.env.OTLP_RATE_LIMIT_WINDOW_MS || '60000', 10); + const max = config?.max ?? parseInt(process.env.OTLP_RATE_LIMIT_MAX || '600', 10); + const isDev = process.env.NODE_ENV === 'development'; + return rateLimit({ + windowMs, + max, + skip: isDev ? () => true : undefined, + standardHeaders: true, + legacyHeaders: false, + message: { error: 'Too many requests to OTLP endpoint, please try again later' }, + }); +} diff --git a/server/src/middleware/trackApiKeyUsage.test.ts b/server/src/middleware/trackApiKeyUsage.test.ts new file mode 100644 index 0000000..ee328c6 --- /dev/null +++ b/server/src/middleware/trackApiKeyUsage.test.ts @@ -0,0 +1,174 @@ +import express from 'express'; +import request from 'supertest'; +import { createTrackApiKeyUsage, incrementRejected, _flush, _accumulator } from './trackApiKeyUsage'; + +// Mock the stores module +const mockBulkUpsert = jest.fn(); +jest.mock('../stores', () => ({ + getStores: () => ({ + apiKeyUsage: { + bulkUpsert: mockBulkUpsert, + }, + }), +})); + +function createApp() { + const app = express(); + app.use(express.json()); + + // Inject apiKeyId + app.use((req, _res, next) => { + req.apiKeyId = 'key-1'; + next(); + }); + + app.use(createTrackApiKeyUsage()); + + app.post('/v1/metrics', (_req, res) => { + res.json({ ok: true }); + }); + + return app; +} + +describe('trackApiKeyUsage', () => { + beforeEach(() => { + jest.clearAllMocks(); + // Clear the accumulator before each test + _accumulator.clear(); + }); + + describe('accumulator increment (DPS-100h)', () => { + it('should increment count in accumulator for both minute and hour granularities', async () => { + const app = createApp(); + + await request(app).post('/v1/metrics').send({}); + + // Should have two entries: one minute, one hour + expect(_accumulator.size).toBe(2); + + const entries = Array.from(_accumulator.entries()); + const granularities = entries.map(([key]) => key.split('|')[1]); + expect(granularities).toContain('minute'); + expect(granularities).toContain('hour'); + + // Both should have count = 1, rejected = 0 + for (const [, val] of entries) { + expect(val.count).toBe(1); + expect(val.rejected).toBe(0); + } + }); + + it('should accumulate multiple requests for the same bucket', async () => { + const app = createApp(); + + await request(app).post('/v1/metrics').send({}); + await request(app).post('/v1/metrics').send({}); + await request(app).post('/v1/metrics').send({}); + + // Still 2 entries (minute + hour), but counts incremented + expect(_accumulator.size).toBe(2); + + for (const [, val] of _accumulator) { + expect(val.count).toBe(3); + expect(val.rejected).toBe(0); + } + }); + }); + + describe('flush (DPS-100h)', () => { + it('should call bulkUpsert with correct entries after flush', async () => { + const app = createApp(); + + await request(app).post('/v1/metrics').send({}); + await request(app).post('/v1/metrics').send({}); + + _flush(); + + expect(mockBulkUpsert).toHaveBeenCalledTimes(1); + + const entries = mockBulkUpsert.mock.calls[0][0]; + expect(entries).toHaveLength(2); // minute + hour + + for (const entry of entries) { + expect(entry.api_key_id).toBe('key-1'); + expect(entry.push_count).toBe(2); + expect(entry.rejected_count).toBe(0); + expect(['minute', 'hour']).toContain(entry.granularity); + expect(entry.bucket_start).toBeDefined(); + } + }); + + it('should clear the accumulator after flush', async () => { + const app = createApp(); + + await request(app).post('/v1/metrics').send({}); + expect(_accumulator.size).toBe(2); + + _flush(); + + expect(_accumulator.size).toBe(0); + }); + + it('should not call bulkUpsert when accumulator is empty', () => { + expect(_accumulator.size).toBe(0); + + _flush(); + + expect(mockBulkUpsert).not.toHaveBeenCalled(); + }); + }); + + describe('incrementRejected (DPS-100i)', () => { + it('should increment rejected count for both minute and hour granularities', () => { + incrementRejected('key-1'); + + expect(_accumulator.size).toBe(2); + + for (const [, val] of _accumulator) { + expect(val.rejected).toBe(1); + expect(val.count).toBe(0); + } + }); + + it('should not affect push count when incrementing rejected', () => { + incrementRejected('key-1'); + incrementRejected('key-1'); + + for (const [, val] of _accumulator) { + expect(val.rejected).toBe(2); + expect(val.count).toBe(0); + } + }); + + it('should track separate keys independently', () => { + incrementRejected('key-1'); + incrementRejected('key-2'); + + // 4 entries: minute + hour for key-1, minute + hour for key-2 + expect(_accumulator.size).toBe(4); + + const key1Entries = Array.from(_accumulator.entries()).filter(([k]) => k.startsWith('key-1')); + const key2Entries = Array.from(_accumulator.entries()).filter(([k]) => k.startsWith('key-2')); + + expect(key1Entries).toHaveLength(2); + expect(key2Entries).toHaveLength(2); + }); + + it('should flush rejected counts correctly via bulkUpsert', () => { + incrementRejected('key-1'); + incrementRejected('key-1'); + + _flush(); + + expect(mockBulkUpsert).toHaveBeenCalledTimes(1); + const entries = mockBulkUpsert.mock.calls[0][0]; + + for (const entry of entries) { + expect(entry.api_key_id).toBe('key-1'); + expect(entry.push_count).toBe(0); + expect(entry.rejected_count).toBe(2); + } + }); + }); +}); diff --git a/server/src/middleware/trackApiKeyUsage.ts b/server/src/middleware/trackApiKeyUsage.ts new file mode 100644 index 0000000..bfdbf15 --- /dev/null +++ b/server/src/middleware/trackApiKeyUsage.ts @@ -0,0 +1,114 @@ +import { Request, Response, NextFunction, RequestHandler } from 'express'; +import { getStores } from '../stores'; +import { BulkUpsertEntry } from '../stores/interfaces/IApiKeyUsageStore'; + +interface AccumulatorEntry { + count: number; + rejected: number; +} + +// Key format: `${apiKeyId}|${granularity}|${bucketStart}` (pipe delimiter avoids ISO colon ambiguity) +let accumulator = new Map(); + +function getMinuteBucketStart(now: Date): string { + return now.toISOString().slice(0, 16) + ':00'; // "2025-01-15T14:32:00" +} + +function getHourBucketStart(now: Date): string { + return now.toISOString().slice(0, 13) + ':00:00'; // "2025-01-15T14:00:00" +} + +export function incrementRejected(apiKeyId: string): void { + const now = new Date(); + for (const [granularity, bucketStart] of [ + ['minute', getMinuteBucketStart(now)], + ['hour', getHourBucketStart(now)], + ] as const) { + const key = `${apiKeyId}|${granularity}|${bucketStart}`; + const entry = accumulator.get(key) ?? { count: 0, rejected: 0 }; + entry.rejected += 1; + accumulator.set(key, entry); + } +} + +function flush(): void { + if (accumulator.size === 0) return; + const snapshot = accumulator; + accumulator = new Map(); // atomic swap — safe in Node's single-threaded event loop + + const entries: BulkUpsertEntry[] = []; + for (const [key, val] of snapshot) { + const pipeIdx1 = key.indexOf('|'); + const pipeIdx2 = key.indexOf('|', pipeIdx1 + 1); + const apiKeyId = key.slice(0, pipeIdx1); + const granularity = key.slice(pipeIdx1 + 1, pipeIdx2) as 'minute' | 'hour'; + const bucketStart = key.slice(pipeIdx2 + 1); + entries.push({ + api_key_id: apiKeyId, + granularity, + bucket_start: bucketStart, + push_count: val.count, + rejected_count: val.rejected, + }); + } + + getStores().apiKeyUsage.bulkUpsert(entries); +} + +let flushInterval: NodeJS.Timeout | null = null; + +/** + * Start the background flusher that periodically persists accumulated usage to the DB. + * Must be called explicitly from the server entry point — not on module load — so that + * importing this module in tests does not leak timers or schedule DB writes. + */ +export function startUsageFlusher(): void { + if (flushInterval) return; + const intervalMs = parseInt(process.env.OTLP_USAGE_FLUSH_INTERVAL_MS ?? '5000', 10); + flushInterval = setInterval(flush, intervalMs); + flushInterval.unref(); + process.on('beforeExit', flushOnExit); +} + +/** Stop the background flusher and perform a final flush. */ +export function stopUsageFlusher(): void { + if (flushInterval) { + clearInterval(flushInterval); + flushInterval = null; + } + process.off('beforeExit', flushOnExit); +} + +function flushOnExit(): void { + if (flushInterval) { + clearInterval(flushInterval); + flushInterval = null; + } + flush(); +} + +export function createTrackApiKeyUsage(): RequestHandler { + return (req: Request, _res: Response, next: NextFunction): void => { + const apiKeyId = req.apiKeyId; + if (!apiKeyId) { + next(); + return; + } + + const now = new Date(); + for (const [granularity, bucketStart] of [ + ['minute', getMinuteBucketStart(now)], + ['hour', getHourBucketStart(now)], + ] as const) { + const key = `${apiKeyId}|${granularity}|${bucketStart}`; + const entry = accumulator.get(key) ?? { count: 0, rejected: 0 }; + entry.count += 1; + accumulator.set(key, entry); + } + + next(); + }; +} + +// Export for testing +export { flush as _flush, accumulator as _accumulator }; diff --git a/server/src/routes/admin/apiKeyRateLimit.test.ts b/server/src/routes/admin/apiKeyRateLimit.test.ts new file mode 100644 index 0000000..3d2f961 --- /dev/null +++ b/server/src/routes/admin/apiKeyRateLimit.test.ts @@ -0,0 +1,413 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +interface TestUser { + id: string; + email: string; + name: string; + role: string; +} + +const adminUser: TestUser = { + id: 'admin-1', + email: 'admin@test.com', + name: 'Admin User', + role: 'admin', +}; + +const regularUser: TestUser = { + id: 'user-1', + email: 'user@test.com', + name: 'Regular User', + role: 'user', +}; + +let currentUser: TestUser = adminUser; + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireAdmin: jest.fn( + ( + req: Record, + res: { status: (code: number) => { json: (body: unknown) => void } }, + next: () => void, + ) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + + if (currentUser.role !== 'admin') { + res.status(403).json({ error: 'Admin access required' }); + return; + } + next(); + }, + ), + requireTeamLead: jest.fn( + ( + req: Record, + res: { status: (code: number) => { json: (body: unknown) => void } }, + next: () => void, + ) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + + if (currentUser.role === 'admin') { + next(); + return; + } + + const teamId = (req.params as Record).id; + const membership = testDb + .prepare('SELECT * FROM team_members WHERE team_id = ? AND user_id = ?') + .get(teamId, currentUser.id) as { role: string } | undefined; + if (!membership || membership.role !== 'lead') { + res.status(403).json({ error: 'Access denied' }); + return; + } + next(); + }, + ), +})); + +import adminRouter from './index'; +import teamRouter from '../teams/index'; + +const app = express(); +app.use(express.json()); +app.use('/api/admin', adminRouter); +app.use('/api/teams', teamRouter); + +describe('Admin API Key Rate Limit Routes', () => { + const teamId = 'team-1'; + + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE team_members ( + team_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (team_id, user_id), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT NOT NULL DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms REAL, + contact TEXT, + contact_override TEXT, + impact_override TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + last_checked TEXT, + last_status_change TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE service_poll_history ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + error TEXT, + recorded_at TEXT NOT NULL, + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + `); + + const insertUser = testDb.prepare( + 'INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)', + ); + insertUser.run(adminUser.id, adminUser.email, adminUser.name, adminUser.role); + insertUser.run(regularUser.id, regularUser.email, regularUser.name, regularUser.role); + + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(teamId, 'Test Team', 'TEST'); + + // Regular user is a lead on team-1 (for lock propagation tests) + testDb.prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)').run(teamId, regularUser.id, 'lead'); + }); + + afterAll(() => { + testDb.close(); + }); + + beforeEach(() => { + currentUser = adminUser; + testDb.exec('DELETE FROM team_api_keys'); + }); + + function insertApiKey( + id: string, + opts: { rate_limit_rpm?: number | null; rate_limit_admin_locked?: number } = {}, + ) { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, rate_limit_admin_locked, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)', + ).run( + id, teamId, `Key ${id}`, `hash-${id}`, 'dps_test', + opts.rate_limit_rpm ?? null, + opts.rate_limit_admin_locked ?? 0, + '2026-01-01T00:00:00Z', + ); + } + + describe('PATCH /api/admin/api-keys/:keyId/rate-limit', () => { + it('should allow admin to set rate_limit_rpm to 0 (unlimited)', async () => { + insertApiKey('key-1'); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: 0 }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_rpm).toBe(0); + expect(res.body.key_hash).toBeUndefined(); + }); + + it('should allow admin to set a custom rate limit', async () => { + insertApiKey('key-1'); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: 50000 }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_rpm).toBe(50000); + }); + + it('should allow admin to reset rate limit to null (default)', async () => { + insertApiKey('key-1', { rate_limit_rpm: 5000 }); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: null }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_rpm).toBeNull(); + }); + + it('should allow admin to lock a key', async () => { + insertApiKey('key-1'); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: 1000, admin_locked: true }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_admin_locked).toBe(1); + expect(res.body.rate_limit_rpm).toBe(1000); + }); + + it('should allow admin to unlock a key', async () => { + insertApiKey('key-1', { rate_limit_admin_locked: 1, rate_limit_rpm: 1000 }); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ admin_locked: false }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_admin_locked).toBe(0); + // rate_limit_rpm should remain unchanged + expect(res.body.rate_limit_rpm).toBe(1000); + }); + + it('should return 404 for nonexistent key', async () => { + const res = await request(app) + .patch('/api/admin/api-keys/nonexistent/rate-limit') + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(404); + expect(res.body.error).toBe('API key not found'); + }); + + it('should return 400 for negative rate_limit_rpm', async () => { + insertApiKey('key-1'); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: -100 }); + + expect(res.status).toBe(400); + expect(res.body.error).toContain('non-negative integer or null'); + }); + + it('should return 400 for non-integer rate_limit_rpm', async () => { + insertApiKey('key-1'); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: 3.14 }); + + expect(res.status).toBe(400); + }); + + it('should deny non-admin users', async () => { + currentUser = regularUser; + insertApiKey('key-1'); + + const res = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(403); + }); + + it('should block team PATCH after admin locks a key', async () => { + // Admin locks the key + insertApiKey('key-1'); + + const lockRes = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ rate_limit_rpm: 1000, admin_locked: true }); + + expect(lockRes.status).toBe(200); + expect(lockRes.body.rate_limit_admin_locked).toBe(1); + + // Team lead tries to update the same key + currentUser = regularUser; + + const teamRes = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(teamRes.status).toBe(403); + expect(teamRes.body.error).toBe('Rate limit locked by admin'); + }); + + it('should allow team PATCH after admin unlocks a key', async () => { + // Start locked + insertApiKey('key-1', { rate_limit_admin_locked: 1, rate_limit_rpm: 1000 }); + + // Admin unlocks + const unlockRes = await request(app) + .patch('/api/admin/api-keys/key-1/rate-limit') + .send({ admin_locked: false }); + + expect(unlockRes.status).toBe(200); + expect(unlockRes.body.rate_limit_admin_locked).toBe(0); + + // Team lead can now update + currentUser = regularUser; + + const teamRes = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(teamRes.status).toBe(200); + expect(teamRes.body.rate_limit_rpm).toBe(5000); + }); + }); +}); diff --git a/server/src/routes/admin/apiKeyRateLimit.ts b/server/src/routes/admin/apiKeyRateLimit.ts new file mode 100644 index 0000000..f8d5db5 --- /dev/null +++ b/server/src/routes/admin/apiKeyRateLimit.ts @@ -0,0 +1,45 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; +import { evictBucket } from '../../middleware/perKeyRateLimit'; + +/** + * PATCH /api/admin/api-keys/:keyId/rate-limit + * Admin endpoint to update any key's rate limit and manage the admin lock. + */ +export function updateAdminApiKeyRateLimit(req: Request, res: Response): void { + try { + const keyId = req.params.keyId; + const stores = getStores(); + + const key = stores.teamApiKeys.findById(keyId); + if (!key) { + res.status(404).json({ error: 'API key not found' }); + return; + } + + const { rate_limit_rpm, admin_locked } = req.body; + + // Validate rate_limit_rpm if provided + if (rate_limit_rpm !== undefined && rate_limit_rpm !== null) { + if (typeof rate_limit_rpm !== 'number' || !Number.isInteger(rate_limit_rpm) || rate_limit_rpm < 0) { + res.status(400).json({ error: 'rate_limit_rpm must be a non-negative integer or null' }); + return; + } + } + + let updated; + if (admin_locked !== undefined) { + updated = stores.teamApiKeys.setAdminLock(keyId, !!admin_locked, rate_limit_rpm); + } else { + updated = stores.teamApiKeys.updateRateLimit(keyId, rate_limit_rpm); + } + + evictBucket(keyId); + + const { key_hash: _hash, ...sanitized } = updated; + res.json(sanitized); + } catch (error) { + sendErrorResponse(res, error, 'updating admin API key rate limit'); + } +} diff --git a/server/src/routes/admin/apiKeyUsage.ts b/server/src/routes/admin/apiKeyUsage.ts new file mode 100644 index 0000000..9d7ea68 --- /dev/null +++ b/server/src/routes/admin/apiKeyUsage.ts @@ -0,0 +1,61 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; + +/** + * GET /api/admin/api-keys/:keyId/usage + * Admin version of the usage time series endpoint; no team membership check. + */ +export function getAdminApiKeyUsage(req: Request, res: Response): void { + try { + const keyId = req.params.keyId; + const stores = getStores(); + + const key = stores.teamApiKeys.findById(keyId); + if (!key) { + res.status(404).json({ error: 'API key not found' }); + return; + } + + const granularity = (req.query.granularity as 'minute' | 'hour') || 'minute'; + if (granularity !== 'minute' && granularity !== 'hour') { + res.status(400).json({ error: 'granularity must be "minute" or "hour"' }); + return; + } + + const now = new Date(); + const defaultFrom = granularity === 'minute' + ? new Date(now.getTime() - 24 * 60 * 60 * 1000) + : new Date(now.getTime() - 30 * 24 * 60 * 60 * 1000); + + const from = (req.query.from as string) || defaultFrom.toISOString(); + const to = (req.query.to as string) || now.toISOString(); + + const buckets = stores.apiKeyUsage.getBuckets(keyId, granularity, from, to); + + res.json({ api_key_id: keyId, granularity, from, to, buckets }); + } catch (error) { + sendErrorResponse(res, error, 'fetching admin API key usage'); + } +} + +/** + * GET /api/admin/otlp-usage + * Cross-team usage overview for the admin dashboard. + */ +export function getAdminOtlpUsage(req: Request, res: Response): void { + try { + const stores = getStores(); + const now = new Date(); + const defaultFrom = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000); + + const from = (req.query.from as string) || defaultFrom.toISOString(); + const to = (req.query.to as string) || now.toISOString(); + + const allBuckets = stores.apiKeyUsage.getAllBuckets('hour', from, to); + + res.json({ from, to, buckets: allBuckets }); + } catch (error) { + sendErrorResponse(res, error, 'fetching admin OTLP usage'); + } +} diff --git a/server/src/routes/admin/index.ts b/server/src/routes/admin/index.ts index cb83573..f609762 100644 --- a/server/src/routes/admin/index.ts +++ b/server/src/routes/admin/index.ts @@ -4,6 +4,10 @@ import { getAuditLog } from './auditLog'; import { getSettings, updateSettings } from './settings'; import { listManifests, syncAllManifests } from './manifests'; import { listAdminAlertMutes } from './alertMutes'; +import { getAdminOtlpStats } from './otlpStats'; +import { updateAdminApiKeyRateLimit } from './apiKeyRateLimit'; +import { getAdminApiKeyUsage, getAdminOtlpUsage } from './apiKeyUsage'; +import { getSpanRetention, updateSpanRetention } from './spanRetention'; const router = Router(); @@ -13,5 +17,11 @@ router.put('/settings', requireAdmin, updateSettings); router.get('/manifests', requireAdmin, listManifests); router.post('/manifests/sync-all', requireAdmin, syncAllManifests); router.get('/alert-mutes', requireAdmin, listAdminAlertMutes); +router.get('/otlp-stats', requireAdmin, getAdminOtlpStats); +router.patch('/api-keys/:keyId/rate-limit', requireAdmin, updateAdminApiKeyRateLimit); +router.get('/api-keys/:keyId/usage', requireAdmin, getAdminApiKeyUsage); +router.get('/otlp-usage', requireAdmin, getAdminOtlpUsage); +router.get('/settings/span-retention', requireAdmin, getSpanRetention); +router.put('/settings/span-retention', requireAdmin, updateSpanRetention); export default router; diff --git a/server/src/routes/admin/otlpStats.test.ts b/server/src/routes/admin/otlpStats.test.ts new file mode 100644 index 0000000..e845762 --- /dev/null +++ b/server/src/routes/admin/otlpStats.test.ts @@ -0,0 +1,312 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((_req, _res, next) => next()), + requireAdmin: jest.fn((_req, _res, next) => next()), +})); + +import adminRouter from './index'; + +const app = express(); +app.use(express.json()); +app.use('/api/admin', adminRouter); + +describe('Admin OTLP Stats API', () => { + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE team_members ( + team_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (team_id, user_id), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT NOT NULL DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms REAL, + contact TEXT, + contact_override TEXT, + impact_override TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + last_checked TEXT, + last_status_change TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE service_poll_history ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + error TEXT, + recorded_at TEXT NOT NULL, + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT + ); + `); + + testDb.prepare('INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)').run('user-1', 'admin@test.com', 'Admin', 'admin'); + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run('team-a', 'Alpha Team', 'ALPHA'); + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run('team-b', 'Beta Team', 'BETA'); + }); + + afterAll(() => { + testDb.close(); + }); + + beforeEach(() => { + testDb.exec('DELETE FROM services'); + testDb.exec('DELETE FROM dependencies'); + testDb.exec('DELETE FROM service_poll_history'); + testDb.exec('DELETE FROM team_api_keys'); + }); + + describe('GET /api/admin/otlp-stats', () => { + it('should return global OTLP stats grouped by team', async () => { + // Alpha: 1 OTLP + 1 default + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-a1', 'Alpha OTLP', 'team-a', 'push://otlp', 'otlp', 1, 1); + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active) VALUES (?, ?, ?, ?, ?, ?)' + ).run('svc-a2', 'Alpha Default', 'team-a', 'http://localhost/health', 'default', 1); + + // Beta: 2 OTLP services + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-b1', 'Beta OTLP 1', 'team-b', 'push://otlp', 'otlp', 1, 1); + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-b2', 'Beta OTLP 2', 'team-b', 'push://otlp', 'otlp', 1, null); + + const res = await request(app).get('/api/admin/otlp-stats'); + + expect(res.status).toBe(200); + expect(res.body.teams).toHaveLength(2); + + const alphaTeam = res.body.teams.find((t: { team_id: string }) => t.team_id === 'team-a'); + const betaTeam = res.body.teams.find((t: { team_id: string }) => t.team_id === 'team-b'); + + expect(alphaTeam.services).toHaveLength(1); + expect(alphaTeam.team_name).toBe('Alpha Team'); + expect(betaTeam.services).toHaveLength(2); + expect(betaTeam.team_name).toBe('Beta Team'); + + expect(res.body.summary.total_otlp_services).toBe(3); + expect(res.body.summary.active_services).toBe(3); + expect(res.body.summary.services_never_pushed).toBe(1); + expect(res.body.summary.total_teams).toBe(2); + }); + + it('should return empty when no OTLP services exist', async () => { + const res = await request(app).get('/api/admin/otlp-stats'); + + expect(res.status).toBe(200); + expect(res.body.teams).toEqual([]); + expect(res.body.summary.total_otlp_services).toBe(0); + expect(res.body.summary.total_teams).toBe(0); + }); + + it('should include API keys per team', async () => { + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-k1', 'Key OTLP', 'team-a', 'push://otlp', 'otlp', 1, 1); + + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_at) VALUES (?, ?, ?, ?, ?, ?)' + ).run('key-a1', 'team-a', 'Alpha Key', 'hash-a1', 'dps_aaaa', '2026-03-15T00:00:00Z'); + + const res = await request(app).get('/api/admin/otlp-stats'); + + expect(res.status).toBe(200); + const alphaTeam = res.body.teams.find((t: { team_id: string }) => t.team_id === 'team-a'); + expect(alphaTeam.apiKeys).toHaveLength(1); + expect(alphaTeam.apiKeys[0].name).toBe('Alpha Key'); + expect(alphaTeam.apiKeys[0].key_prefix).toBe('dps_aaaa'); + expect(alphaTeam.apiKeys[0].key_hash).toBeUndefined(); + }); + + it('should include rate limit and usage fields on API keys', async () => { + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-rl', 'RL OTLP', 'team-a', 'push://otlp', 'otlp', 1, 1); + + // Custom rate limit key + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, rate_limit_admin_locked, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('key-rl', 'team-a', 'RL Key', 'hash-rl', 'dps_rl', 3000, 1, '2026-03-15T00:00:00Z'); + + // Insert usage data + const now = new Date(); + const recentBucket = new Date(now.getTime() - 30 * 60 * 1000).toISOString().slice(0, 13) + ':00:00'; + testDb.prepare( + 'INSERT INTO api_key_usage_buckets (api_key_id, bucket_start, granularity, push_count, rejected_count) VALUES (?, ?, ?, ?, ?)' + ).run('key-rl', recentBucket, 'hour', 500, 20); + + const res = await request(app).get('/api/admin/otlp-stats'); + + expect(res.status).toBe(200); + const alphaTeam = res.body.teams.find((t: { team_id: string }) => t.team_id === 'team-a'); + expect(alphaTeam.apiKeys).toHaveLength(1); + + const key = alphaTeam.apiKeys[0]; + expect(key.rate_limit_rpm).toBe(3000); + expect(key.rate_limit_is_custom).toBe(true); + expect(key.rate_limit_admin_locked).toBe(true); + expect(key.usage_1h).toBeGreaterThanOrEqual(0); + expect(key.usage_24h).toBeGreaterThanOrEqual(500); + expect(key.usage_7d).toBeGreaterThanOrEqual(500); + expect(key.rejected_24h).toBeGreaterThanOrEqual(20); + expect(key.rejected_7d).toBeGreaterThanOrEqual(20); + }); + + it('should use default rate_limit_rpm when null in DB', async () => { + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-def', 'Default OTLP', 'team-b', 'push://otlp', 'otlp', 1, 1); + + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('key-def', 'team-b', 'Default Key', 'hash-def', 'dps_def', null, '2026-03-15T00:00:00Z'); + + const res = await request(app).get('/api/admin/otlp-stats'); + + expect(res.status).toBe(200); + const betaTeam = res.body.teams.find((t: { team_id: string }) => t.team_id === 'team-b'); + const key = betaTeam.apiKeys[0]; + expect(key.rate_limit_rpm).toBe(150000); + expect(key.rate_limit_is_custom).toBe(false); + }); + + it('should count error services correctly across teams', async () => { + // Alpha: 1 healthy OTLP + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-e1', 'Healthy', 'team-a', 'push://otlp', 'otlp', 1, 1); + + // Beta: 1 error OTLP + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success, last_poll_error) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('svc-e2', 'Failing', 'team-b', 'push://otlp', 'otlp', 1, 0, 'timeout'); + + const res = await request(app).get('/api/admin/otlp-stats'); + + expect(res.status).toBe(200); + expect(res.body.summary.services_with_errors).toBe(1); + expect(res.body.summary.total_otlp_services).toBe(2); + }); + }); +}); diff --git a/server/src/routes/admin/otlpStats.ts b/server/src/routes/admin/otlpStats.ts new file mode 100644 index 0000000..fbc1d0b --- /dev/null +++ b/server/src/routes/admin/otlpStats.ts @@ -0,0 +1,94 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; + +export function getAdminOtlpStats(req: Request, res: Response): void { + try { + const stores = getStores(); + + const allServices = stores.services.findAll(); + const otlpServices = allServices.filter(s => s.health_endpoint_format === 'otlp'); + + // Collect all key IDs across all teams for a single batch summary query + const allTeamIds = [...new Set(otlpServices.map(s => s.team_id))]; + const allApiKeys = allTeamIds.flatMap(tid => stores.teamApiKeys.findByTeamId(tid)); + const allKeyIds = allApiKeys.map(k => k.id); + const now = new Date().toISOString(); + const minus1h = new Date(Date.now() - 60 * 60 * 1000).toISOString(); + const minus24h = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); + const minus7d = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString(); + const summaries1h = stores.apiKeyUsage.getSummaryForKeys(allKeyIds, minus1h, now); + const summaries24h = stores.apiKeyUsage.getSummaryForKeys(allKeyIds, minus24h, now); + const summaries7d = stores.apiKeyUsage.getSummaryForKeys(allKeyIds, minus7d, now); + const defaultRpm = parseInt(process.env.OTLP_PER_KEY_RATE_LIMIT_RPM ?? '150000', 10); + + // Group by team + const teamIds = allTeamIds; + + const teams = teamIds.map(teamId => { + const team = stores.teams.findById(teamId); + const teamOtlpServices = otlpServices.filter(s => s.team_id === teamId); + const apiKeys = stores.teamApiKeys.findByTeamId(teamId); + + const services = teamOtlpServices.map(s => { + const depCount = stores.dependencies.findByServiceId(s.id).length; + const errors24h = stores.servicePollHistory.getErrorCount24h(s.id); + + let parsedWarnings: string[] | null = null; + if (s.poll_warnings) { + try { + parsedWarnings = JSON.parse(s.poll_warnings); + } catch { + parsedWarnings = null; + } + } + + return { + id: s.id, + name: s.name, + is_active: s.is_active, + last_push_success: s.last_poll_success, + last_push_error: s.last_poll_error, + last_push_warnings: parsedWarnings, + last_push_at: s.updated_at, + dependency_count: depCount, + errors_24h: errors24h, + schema_config: s.schema_config, + }; + }); + + return { + team_id: teamId, + team_name: team?.name ?? 'Unknown', + services, + apiKeys: apiKeys.map(k => ({ + id: k.id, + name: k.name, + key_prefix: k.key_prefix, + last_used_at: k.last_used_at, + created_at: k.created_at, + rate_limit_rpm: k.rate_limit_rpm ?? defaultRpm, + rate_limit_is_custom: k.rate_limit_rpm !== null, + rate_limit_admin_locked: Boolean(k.rate_limit_admin_locked), + usage_1h: summaries1h.get(k.id)?.push_count ?? 0, + usage_24h: summaries24h.get(k.id)?.push_count ?? 0, + usage_7d: summaries7d.get(k.id)?.push_count ?? 0, + rejected_24h: summaries24h.get(k.id)?.rejected_count ?? 0, + rejected_7d: summaries7d.get(k.id)?.rejected_count ?? 0, + })), + }; + }); + + const summary = { + total_otlp_services: otlpServices.length, + active_services: otlpServices.filter(s => s.is_active).length, + services_with_errors: otlpServices.filter(s => s.last_poll_success === 0).length, + services_never_pushed: otlpServices.filter(s => s.last_poll_success === null).length, + total_teams: teamIds.length, + }; + + res.json({ teams, summary }); + } catch (error) { + sendErrorResponse(res, error, 'getting admin OTLP stats'); + } +} diff --git a/server/src/routes/admin/spanRetention.test.ts b/server/src/routes/admin/spanRetention.test.ts new file mode 100644 index 0000000..73d173d --- /dev/null +++ b/server/src/routes/admin/spanRetention.test.ts @@ -0,0 +1,161 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +const defaultAdminUser = { + id: 'user-1', + email: 'admin@test.com', + name: 'Admin User', + role: 'admin', + is_active: 1, +}; + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((req, _res, next) => { + req.user = defaultAdminUser; + next(); + }), + requireAdmin: jest.fn((_req, _res, next) => next()), +})); + +jest.mock('../../services/audit/AuditLogService', () => ({ + auditFromRequest: jest.fn(), +})); + +const mockAppSettingsGet = jest.fn(); +const mockAppSettingsSet = jest.fn(); + +jest.mock('../../stores', () => ({ + getStores: () => ({ + appSettings: { + get: mockAppSettingsGet, + set: mockAppSettingsSet, + }, + }), +})); + +import adminRouter from './index'; +import { auditFromRequest } from '../../services/audit/AuditLogService'; + +const app = express(); +app.use(express.json()); +app.use((req, _res, next) => { + req.user = defaultAdminUser as never; + next(); +}); +app.use('/api/admin', adminRouter); + +describe('Admin Span Retention API', () => { + beforeEach(() => { + jest.clearAllMocks(); + mockAppSettingsGet.mockReturnValue('7'); + }); + + describe('GET /api/admin/settings/span-retention', () => { + it('should return current span retention days', async () => { + mockAppSettingsGet.mockReturnValue('14'); + + const response = await request(app).get('/api/admin/settings/span-retention'); + + expect(response.status).toBe(200); + expect(response.body.days).toBe(14); + expect(mockAppSettingsGet).toHaveBeenCalledWith('span_retention_days'); + }); + + it('should return default 7 days when no setting exists', async () => { + mockAppSettingsGet.mockReturnValue(undefined); + + const response = await request(app).get('/api/admin/settings/span-retention'); + + expect(response.status).toBe(200); + expect(response.body.days).toBe(7); + }); + }); + + describe('PUT /api/admin/settings/span-retention', () => { + it('should update span retention days', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 14 }); + + expect(response.status).toBe(200); + expect(response.body.days).toBe(14); + expect(mockAppSettingsSet).toHaveBeenCalledWith('span_retention_days', '14', 'user-1'); + }); + + it('should audit the settings change', async () => { + await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 30 }); + + expect(auditFromRequest).toHaveBeenCalledWith( + expect.anything(), + 'settings.updated', + 'settings', + undefined, + { key: 'span_retention_days', value: 30 }, + ); + }); + + it('should return 400 for non-integer days', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 3.5 }); + + expect(response.status).toBe(400); + expect(response.body.error).toBe('days must be an integer'); + }); + + it('should return 400 for non-number days', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 'seven' }); + + expect(response.status).toBe(400); + expect(response.body.error).toBe('days must be an integer'); + }); + + it('should return 400 for days below minimum (1)', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 0 }); + + expect(response.status).toBe(400); + expect(response.body.error).toContain('between 1 and 365'); + }); + + it('should return 400 for days above maximum (365)', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 400 }); + + expect(response.status).toBe(400); + expect(response.body.error).toContain('between 1 and 365'); + }); + + it('should accept minimum value (1)', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 1 }); + + expect(response.status).toBe(200); + expect(response.body.days).toBe(1); + }); + + it('should accept maximum value (365)', async () => { + const response = await request(app) + .put('/api/admin/settings/span-retention') + .send({ days: 365 }); + + expect(response.status).toBe(200); + expect(response.body.days).toBe(365); + }); + }); +}); diff --git a/server/src/routes/admin/spanRetention.ts b/server/src/routes/admin/spanRetention.ts new file mode 100644 index 0000000..75a02ee --- /dev/null +++ b/server/src/routes/admin/spanRetention.ts @@ -0,0 +1,50 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; +import { auditFromRequest } from '../../services/audit/AuditLogService'; + +const DEFAULT_SPAN_RETENTION_DAYS = 7; +const MIN_SPAN_RETENTION_DAYS = 1; +const MAX_SPAN_RETENTION_DAYS = 365; + +export function getSpanRetention(_req: Request, res: Response): void { + try { + const stores = getStores(); + const value = stores.appSettings.get('span_retention_days'); + const days = value ? parseInt(value, 10) : DEFAULT_SPAN_RETENTION_DAYS; + + res.json({ days }); + } catch (error) { + sendErrorResponse(res, error, 'fetching span retention setting'); + } +} + +export function updateSpanRetention(req: Request, res: Response): void { + try { + const { days } = req.body; + + if (typeof days !== 'number' || !Number.isInteger(days)) { + res.status(400).json({ error: 'days must be an integer' }); + return; + } + + if (days < MIN_SPAN_RETENTION_DAYS || days > MAX_SPAN_RETENTION_DAYS) { + res.status(400).json({ + error: `days must be between ${MIN_SPAN_RETENTION_DAYS} and ${MAX_SPAN_RETENTION_DAYS}`, + }); + return; + } + + const stores = getStores(); + stores.appSettings.set('span_retention_days', String(days), req.user!.id); + + auditFromRequest(req, 'settings.updated', 'settings', undefined, { + key: 'span_retention_days', + value: days, + }); + + res.json({ days }); + } catch (error) { + sendErrorResponse(res, error, 'updating span retention setting'); + } +} diff --git a/server/src/routes/alerts/channels/maskConfig.ts b/server/src/routes/alerts/channels/maskConfig.ts index 8374ea6..5d36b45 100644 --- a/server/src/routes/alerts/channels/maskConfig.ts +++ b/server/src/routes/alerts/channels/maskConfig.ts @@ -43,7 +43,7 @@ function maskWebhookConfig(config: WebhookConfig): WebhookConfig { if (config.headers) { const maskedHeaders: Record = {}; for (const key of Object.keys(config.headers)) { - maskedHeaders[key] = MASK; + maskedHeaders[key] = MASK; // eslint-disable-line security/detect-object-injection } result.headers = maskedHeaders; } diff --git a/server/src/routes/associations/associations-auth.test.ts b/server/src/routes/associations/associations-auth.test.ts index 3338fbe..a31fd68 100644 --- a/server/src/routes/associations/associations-auth.test.ts +++ b/server/src/routes/associations/associations-auth.test.ts @@ -128,6 +128,10 @@ describe('Associations API - Authorization (IDOR)', () => { last_check_at TEXT, check_details TEXT, skipped INTEGER NOT NULL DEFAULT 0, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')), FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE, @@ -141,6 +145,8 @@ describe('Associations API - Authorization (IDOR)', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT NOT NULL DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE, diff --git a/server/src/routes/associations/associations.test.ts b/server/src/routes/associations/associations.test.ts index 6e30870..e71aed0 100644 --- a/server/src/routes/associations/associations.test.ts +++ b/server/src/routes/associations/associations.test.ts @@ -102,6 +102,10 @@ describe('Associations API', () => { last_check_at TEXT, check_details TEXT, skipped INTEGER NOT NULL DEFAULT 0, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')), FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE, @@ -115,6 +119,8 @@ describe('Associations API', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT NOT NULL DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE, @@ -310,4 +316,107 @@ describe('Associations API', () => { expect(remaining).toBeUndefined(); }); }); + + describe('PUT /api/dependencies/:depId/associations/:assocId/confirm', () => { + it('should confirm an auto-suggested association', async () => { + associationId = randomUUID(); + testDb.prepare(` + INSERT INTO dependency_associations (id, dependency_id, linked_service_id, association_type, is_auto_suggested) + VALUES (?, ?, ?, ?, 1) + `).run(associationId, dependencyId, linkedServiceId, 'api_call'); + + const response = await request(app) + .put(`/api/dependencies/${dependencyId}/associations/${associationId}/confirm`); + + expect(response.status).toBe(200); + expect(response.body.success).toBe(true); + + // Verify is_auto_suggested is now 0 + const row = testDb.prepare('SELECT is_auto_suggested FROM dependency_associations WHERE id = ?').get(associationId) as { is_auto_suggested: number }; + expect(row.is_auto_suggested).toBe(0); + }); + + it('should return 404 for non-existent dependency', async () => { + const response = await request(app) + .put(`/api/dependencies/non-existent/associations/non-existent/confirm`); + + expect(response.status).toBe(404); + }); + + it('should return 404 for non-existent association', async () => { + const response = await request(app) + .put(`/api/dependencies/${dependencyId}/associations/non-existent/confirm`); + + expect(response.status).toBe(404); + expect(response.body.error).toBe('Association not found'); + }); + + it('should return 404 when association belongs to different dependency', async () => { + // Create another dependency + const otherDepId = randomUUID(); + testDb.prepare('INSERT INTO dependencies (id, service_id, name, status) VALUES (?, ?, ?, ?)') + .run(otherDepId, serviceId, 'other-dep', 'healthy'); + + associationId = randomUUID(); + testDb.prepare(` + INSERT INTO dependency_associations (id, dependency_id, linked_service_id, association_type, is_auto_suggested) + VALUES (?, ?, ?, ?, 1) + `).run(associationId, otherDepId, linkedServiceId, 'api_call'); + + const response = await request(app) + .put(`/api/dependencies/${dependencyId}/associations/${associationId}/confirm`); + + expect(response.status).toBe(404); + expect(response.body.error).toBe('Association not found'); + }); + + it('should succeed for already-confirmed association', async () => { + associationId = randomUUID(); + testDb.prepare(` + INSERT INTO dependency_associations (id, dependency_id, linked_service_id, association_type, is_auto_suggested) + VALUES (?, ?, ?, ?, 0) + `).run(associationId, dependencyId, linkedServiceId, 'api_call'); + + const response = await request(app) + .put(`/api/dependencies/${dependencyId}/associations/${associationId}/confirm`); + + expect(response.status).toBe(200); + expect(response.body.success).toBe(true); + }); + }); + + describe('PUT /api/dependencies/:depId/associations/:assocId/dismiss', () => { + it('should dismiss an auto-suggested association', async () => { + associationId = randomUUID(); + testDb.prepare(` + INSERT INTO dependency_associations (id, dependency_id, linked_service_id, association_type, is_auto_suggested) + VALUES (?, ?, ?, ?, 1) + `).run(associationId, dependencyId, linkedServiceId, 'api_call'); + + const response = await request(app) + .put(`/api/dependencies/${dependencyId}/associations/${associationId}/dismiss`); + + expect(response.status).toBe(200); + expect(response.body.success).toBe(true); + + // Verify is_dismissed is now 1 + const row = testDb.prepare('SELECT is_dismissed FROM dependency_associations WHERE id = ?').get(associationId) as { is_dismissed: number }; + expect(row.is_dismissed).toBe(1); + }); + + it('should return 404 for non-existent dependency', async () => { + const response = await request(app) + .put(`/api/dependencies/non-existent/associations/non-existent/dismiss`); + + expect(response.status).toBe(404); + }); + + it('should return 404 for non-existent association', async () => { + const response = await request(app) + .put(`/api/dependencies/${dependencyId}/associations/non-existent/dismiss`); + + expect(response.status).toBe(404); + expect(response.body.error).toBe('Association not found'); + }); + }); }); diff --git a/server/src/routes/associations/confirmAssociation.ts b/server/src/routes/associations/confirmAssociation.ts new file mode 100644 index 0000000..48931d6 --- /dev/null +++ b/server/src/routes/associations/confirmAssociation.ts @@ -0,0 +1,31 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { AuthorizationService } from '../../auth/authorizationService'; +import { sendErrorResponse } from '../../utils/errors'; + +export function confirmAssociation(req: Request, res: Response): void { + try { + const { depId, assocId } = req.params; + const stores = getStores(); + + // Verify user has team access to the dependency's owning service + const authResult = AuthorizationService.checkDependencyTeamAccess(req.user!, depId); + if (!authResult.authorized) { + res.status(authResult.statusCode!).json({ error: authResult.error }); + return; + } + + // Verify association exists and belongs to this dependency + const association = stores.associations.findById(assocId); + if (!association || association.dependency_id !== depId) { + res.status(404).json({ error: 'Association not found' }); + return; + } + + stores.associations.confirm(assocId); + + res.status(200).json({ success: true }); + } catch (error) /* istanbul ignore next -- Catch block for unexpected database/infrastructure errors */ { + sendErrorResponse(res, error, 'confirming association'); + } +} diff --git a/server/src/routes/associations/dismissAssociation.ts b/server/src/routes/associations/dismissAssociation.ts new file mode 100644 index 0000000..94fbe9a --- /dev/null +++ b/server/src/routes/associations/dismissAssociation.ts @@ -0,0 +1,31 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { AuthorizationService } from '../../auth/authorizationService'; +import { sendErrorResponse } from '../../utils/errors'; + +export function dismissAssociation(req: Request, res: Response): void { + try { + const { depId, assocId } = req.params; + const stores = getStores(); + + // Verify user has team access to the dependency's owning service + const authResult = AuthorizationService.checkDependencyTeamAccess(req.user!, depId); + if (!authResult.authorized) { + res.status(authResult.statusCode!).json({ error: authResult.error }); + return; + } + + // Verify association exists and belongs to this dependency + const association = stores.associations.findById(assocId); + if (!association || association.dependency_id !== depId) { + res.status(404).json({ error: 'Association not found' }); + return; + } + + stores.associations.dismiss(assocId); + + res.status(200).json({ success: true }); + } catch (error) /* istanbul ignore next -- Catch block for unexpected database/infrastructure errors */ { + sendErrorResponse(res, error, 'dismissing association'); + } +} diff --git a/server/src/routes/associations/getAssociations.ts b/server/src/routes/associations/getAssociations.ts index 04e7127..1d2556f 100644 --- a/server/src/routes/associations/getAssociations.ts +++ b/server/src/routes/associations/getAssociations.ts @@ -30,6 +30,8 @@ export function getAssociations(req: Request, res: Response): void { dependency_id: row.dependency_id, linked_service_id: row.linked_service_id, association_type: row.association_type, + is_auto_suggested: row.is_auto_suggested, + is_dismissed: row.is_dismissed, manifest_managed: row.manifest_managed, created_at: row.created_at, linked_service: linkedService!, diff --git a/server/src/routes/associations/index.ts b/server/src/routes/associations/index.ts index b6cce31..a9d4aa3 100644 --- a/server/src/routes/associations/index.ts +++ b/server/src/routes/associations/index.ts @@ -2,6 +2,8 @@ import { Router } from 'express'; import { getAssociations } from './getAssociations'; import { createAssociation } from './createAssociation'; import { deleteAssociation } from './deleteAssociation'; +import { confirmAssociation } from './confirmAssociation'; +import { dismissAssociation } from './dismissAssociation'; const router = Router(); @@ -15,4 +17,10 @@ router.post('/dependencies/:dependencyId/associations', createAssociation); // DELETE /api/dependencies/:dependencyId/associations/:serviceId - Remove an association router.delete('/dependencies/:dependencyId/associations/:serviceId', deleteAssociation); +// PUT /api/dependencies/:depId/associations/:assocId/confirm - Confirm an auto-suggested association +router.put('/dependencies/:depId/associations/:assocId/confirm', confirmAssociation); + +// PUT /api/dependencies/:depId/associations/:assocId/dismiss - Dismiss an auto-suggested association +router.put('/dependencies/:depId/associations/:assocId/dismiss', dismissAssociation); + export default router; diff --git a/server/src/routes/dependencies/enrichDiscovered.test.ts b/server/src/routes/dependencies/enrichDiscovered.test.ts new file mode 100644 index 0000000..37b62c0 --- /dev/null +++ b/server/src/routes/dependencies/enrichDiscovered.test.ts @@ -0,0 +1,194 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; +import { randomUUID } from 'crypto'; + +// Create in-memory database for testing +const testDb = new Database(':memory:'); + +// Mock the db module +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +// Mock the auth module +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((_req, _res, next) => next()), + requireAdmin: jest.fn((_req, _res, next) => next()), + requireTeamAccess: jest.fn((_req, _res, next) => next()), + requireTeamLead: jest.fn((_req, _res, next) => next()), + requireServiceTeamLead: jest.fn((_req, _res, next) => next()), + requireBodyTeamLead: jest.fn((_req, _res, next) => next()), +})); + +import { StoreRegistry } from '../../stores'; +import dependenciesRouter from './index'; + +const adminUser = { + id: 'admin-test-user-id', + email: 'admin@test.com', + name: 'Admin', + oidc_subject: null, + password_hash: null, + role: 'admin' as const, + is_active: 1, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), +}; + +const app = express(); +app.use(express.json()); +app.use((req, _res, next) => { + req.user = adminUser; + next(); +}); +app.use('/api/dependencies', dependenciesRouter); + +describe('Dependency Enrichment API', () => { + let teamId: string; + let serviceId: string; + let dependencyId: string; + + beforeAll(() => { + testDb.pragma('foreign_keys = ON'); + + testDb.exec(` + CREATE TABLE IF NOT EXISTS teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ) + `); + + testDb.exec(` + CREATE TABLE IF NOT EXISTS services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE RESTRICT + ) + `); + + testDb.exec(` + CREATE TABLE IF NOT EXISTS dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT, + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms REAL, + contact TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, + status TEXT NOT NULL DEFAULT 'unknown', + last_checked TEXT, + last_status_change TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE, + UNIQUE(service_id, name) + ) + `); + }); + + beforeEach(() => { + testDb.exec('DELETE FROM dependencies'); + testDb.exec('DELETE FROM services'); + testDb.exec('DELETE FROM teams'); + StoreRegistry.resetInstance(); + jest.clearAllMocks(); + + teamId = randomUUID(); + testDb.prepare('INSERT INTO teams (id, name) VALUES (?, ?)').run(teamId, 'Test Team'); + + serviceId = randomUUID(); + testDb.prepare('INSERT INTO services (id, name, team_id, health_endpoint) VALUES (?, ?, ?, ?)') + .run(serviceId, 'Test Service', teamId, 'https://example.com/health'); + + dependencyId = randomUUID(); + testDb.prepare(` + INSERT INTO dependencies (id, service_id, name, status, discovery_source) + VALUES (?, ?, ?, ?, ?) + `).run(dependencyId, serviceId, 'postgres-db', 'healthy', 'otlp_trace'); + }); + + afterAll(() => { + testDb.close(); + }); + + describe('PATCH /api/dependencies/:id/enrich', () => { + it('should update user enrichment fields', async () => { + const response = await request(app) + .patch(`/api/dependencies/${dependencyId}/enrich`) + .send({ displayName: 'PostgreSQL Primary', description: 'Main database', impact: 'Critical' }); + + expect(response.status).toBe(200); + expect(response.body.user_display_name).toBe('PostgreSQL Primary'); + expect(response.body.user_description).toBe('Main database'); + expect(response.body.user_impact).toBe('Critical'); + }); + + it('should allow partial update (only displayName)', async () => { + const response = await request(app) + .patch(`/api/dependencies/${dependencyId}/enrich`) + .send({ displayName: 'PostgreSQL Primary' }); + + expect(response.status).toBe(200); + expect(response.body.user_display_name).toBe('PostgreSQL Primary'); + expect(response.body.user_description).toBeNull(); + }); + + it('should persist enrichment across subsequent calls', async () => { + await request(app) + .patch(`/api/dependencies/${dependencyId}/enrich`) + .send({ displayName: 'PostgreSQL Primary', description: 'Main DB' }); + + // Update only impact + const response = await request(app) + .patch(`/api/dependencies/${dependencyId}/enrich`) + .send({ impact: 'Critical' }); + + expect(response.status).toBe(200); + // displayName was set in previous call, should still be there + expect(response.body.user_display_name).toBe('PostgreSQL Primary'); + expect(response.body.user_impact).toBe('Critical'); + }); + + it('should return 404 for non-existent dependency', async () => { + const response = await request(app) + .patch('/api/dependencies/non-existent/enrich') + .send({ displayName: 'Test' }); + + expect(response.status).toBe(404); + }); + + it('should return 400 when no enrichment fields provided', async () => { + const response = await request(app) + .patch(`/api/dependencies/${dependencyId}/enrich`) + .send({}); + + expect(response.status).toBe(400); + expect(response.body.error).toContain('At least one enrichment field'); + }); + }); +}); diff --git a/server/src/routes/dependencies/enrichDiscoveredDep.ts b/server/src/routes/dependencies/enrichDiscoveredDep.ts new file mode 100644 index 0000000..109514e --- /dev/null +++ b/server/src/routes/dependencies/enrichDiscoveredDep.ts @@ -0,0 +1,41 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { AuthorizationService } from '../../auth/authorizationService'; +import { sendErrorResponse } from '../../utils/errors'; + +export function enrichDiscoveredDep(req: Request, res: Response): void { + try { + const { id } = req.params; + const stores = getStores(); + + // Verify user has team access to the dependency's owning service + const authResult = AuthorizationService.checkDependencyTeamAccess(req.user!, id); + if (!authResult.authorized) { + res.status(authResult.statusCode!).json({ error: authResult.error }); + return; + } + + const enrichment: { displayName?: string | null; description?: string | null; impact?: string | null } = {}; + + if ('displayName' in req.body) enrichment.displayName = req.body.displayName; + if ('description' in req.body) enrichment.description = req.body.description; + if ('impact' in req.body) enrichment.impact = req.body.impact; + + // Validate at least one field is provided + if (Object.keys(enrichment).length === 0) { + res.status(400).json({ error: 'At least one enrichment field is required (displayName, description, impact)' }); + return; + } + + const updated = stores.dependencies.updateUserEnrichment(id, enrichment); + + if (!updated) { + res.status(404).json({ error: 'Dependency not found' }); + return; + } + + res.status(200).json(updated); + } catch (error) /* istanbul ignore next -- Catch block for unexpected database/infrastructure errors */ { + sendErrorResponse(res, error, 'enriching dependency'); + } +} diff --git a/server/src/routes/dependencies/index.ts b/server/src/routes/dependencies/index.ts index 6f19b37..6d2bba8 100644 --- a/server/src/routes/dependencies/index.ts +++ b/server/src/routes/dependencies/index.ts @@ -2,6 +2,7 @@ import { Router } from 'express'; import { getHealthTimeline } from './getHealthTimeline'; import { putOverride } from './putOverride'; import { deleteOverride } from './deleteOverride'; +import { enrichDiscoveredDep } from './enrichDiscoveredDep'; const router = Router(); @@ -14,4 +15,7 @@ router.put('/:id/overrides', putOverride); // DELETE /api/dependencies/:id/overrides - Clear all per-instance overrides router.delete('/:id/overrides', deleteOverride); +// PATCH /api/dependencies/:id/enrich - Set user enrichment for a discovered dependency +router.patch('/:id/enrich', enrichDiscoveredDep); + export default router; diff --git a/server/src/routes/dependencies/listDiscovered.ts b/server/src/routes/dependencies/listDiscovered.ts new file mode 100644 index 0000000..e3d860e --- /dev/null +++ b/server/src/routes/dependencies/listDiscovered.ts @@ -0,0 +1,40 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { AuthorizationService } from '../../auth/authorizationService'; +import { sendErrorResponse } from '../../utils/errors'; + +export function listDiscovered(req: Request, res: Response): void { + try { + const { serviceId } = req.params; + const stores = getStores(); + + // Verify service exists + const service = stores.services.findById(serviceId); + if (!service) { + res.status(404).json({ error: 'Service not found' }); + return; + } + + // Verify user has team access + const authResult = AuthorizationService.checkServiceTeamAccess(req.user!, serviceId); + if (!authResult.authorized) { + res.status(authResult.statusCode!).json({ error: authResult.error }); + return; + } + + const dependencies = stores.dependencies.findByDiscoverySource(serviceId, 'otlp_trace'); + + // Attach auto-suggested associations for each dependency + const result = dependencies.map((dep) => { + const autoSuggested = stores.associations.findAutoSuggested(dep.id); + return { + ...dep, + auto_suggested_associations: autoSuggested, + }; + }); + + res.status(200).json(result); + } catch (error) /* istanbul ignore next -- Catch block for unexpected database/infrastructure errors */ { + sendErrorResponse(res, error, 'listing discovered dependencies'); + } +} diff --git a/server/src/routes/external-services/external-services.test.ts b/server/src/routes/external-services/external-services.test.ts index dba325b..0e0eb88 100644 --- a/server/src/routes/external-services/external-services.test.ts +++ b/server/src/routes/external-services/external-services.test.ts @@ -99,6 +99,7 @@ describe('External Services API', () => { is_active INTEGER NOT NULL DEFAULT 1, is_external INTEGER NOT NULL DEFAULT 0, description TEXT, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', last_poll_success INTEGER, last_poll_error TEXT, poll_warnings TEXT, diff --git a/server/src/routes/externalNodes/externalNodes.test.ts b/server/src/routes/externalNodes/externalNodes.test.ts new file mode 100644 index 0000000..65ee2f7 --- /dev/null +++ b/server/src/routes/externalNodes/externalNodes.test.ts @@ -0,0 +1,196 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +// Create in-memory database for testing +const testDb = new Database(':memory:'); + +// Mock the db module +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +// Mock the auth module +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((_req, _res, next) => next()), + requireAdmin: jest.fn((_req, _res, next) => next()), + requireTeamAccess: jest.fn((_req, _res, next) => next()), + requireTeamLead: jest.fn((_req, _res, next) => next()), + requireServiceTeamLead: jest.fn((_req, _res, next) => next()), + requireBodyTeamLead: jest.fn((_req, _res, next) => next()), +})); + +import { StoreRegistry } from '../../stores'; +import externalNodesRouter from './index'; + +const adminUser = { + id: 'admin-test-user-id', + email: 'admin@test.com', + name: 'Admin', + oidc_subject: null, + password_hash: null, + role: 'admin' as const, + is_active: 1, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), +}; + +const app = express(); +app.use(express.json()); +app.use((req, _res, next) => { + req.user = adminUser; + next(); +}); +app.use('/api/external-nodes', externalNodesRouter); + +describe('External Nodes API', () => { + beforeAll(() => { + testDb.pragma('foreign_keys = ON'); + + testDb.exec(` + CREATE TABLE IF NOT EXISTS users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ) + `); + + testDb.exec(` + CREATE TABLE IF NOT EXISTS external_node_enrichment ( + id TEXT PRIMARY KEY, + canonical_name TEXT NOT NULL UNIQUE, + display_name TEXT, + description TEXT, + impact TEXT, + contact TEXT, + service_type TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT, + FOREIGN KEY (updated_by) REFERENCES users(id) + ) + `); + }); + + beforeEach(() => { + testDb.exec('DELETE FROM external_node_enrichment'); + testDb.exec('DELETE FROM users'); + + // Insert admin user so FK constraint on updated_by is satisfied + testDb.prepare('INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)') + .run('admin-test-user-id', 'admin@test.com', 'Admin', 'admin'); + + StoreRegistry.resetInstance(); + jest.clearAllMocks(); + }); + + afterAll(() => { + testDb.close(); + }); + + describe('GET /api/external-nodes', () => { + it('should return empty array when no enrichments exist', async () => { + const response = await request(app).get('/api/external-nodes'); + expect(response.status).toBe(200); + expect(response.body).toEqual([]); + }); + + it('should return all enrichment records', async () => { + // Upsert two enrichments + await request(app) + .put('/api/external-nodes/stripe-api') + .send({ displayName: 'Stripe API', serviceType: 'payment' }); + + await request(app) + .put('/api/external-nodes/postgresql') + .send({ displayName: 'PostgreSQL', serviceType: 'database' }); + + const response = await request(app).get('/api/external-nodes'); + expect(response.status).toBe(200); + expect(response.body).toHaveLength(2); + }); + }); + + describe('PUT /api/external-nodes/:canonicalName', () => { + it('should create new enrichment', async () => { + const response = await request(app) + .put('/api/external-nodes/stripe-api') + .send({ + displayName: 'Stripe API', + description: 'Payment gateway', + impact: 'Revenue-critical', + serviceType: 'payment', + }); + + expect(response.status).toBe(200); + expect(response.body.canonical_name).toBe('stripe-api'); + expect(response.body.display_name).toBe('Stripe API'); + expect(response.body.description).toBe('Payment gateway'); + expect(response.body.impact).toBe('Revenue-critical'); + expect(response.body.service_type).toBe('payment'); + }); + + it('should update existing enrichment', async () => { + // Create first + await request(app) + .put('/api/external-nodes/stripe-api') + .send({ displayName: 'Stripe' }); + + // Update + const response = await request(app) + .put('/api/external-nodes/stripe-api') + .send({ displayName: 'Stripe API v2', description: 'Updated description' }); + + expect(response.status).toBe(200); + expect(response.body.display_name).toBe('Stripe API v2'); + expect(response.body.description).toBe('Updated description'); + + // Verify only one record exists + const listResponse = await request(app).get('/api/external-nodes'); + expect(listResponse.body).toHaveLength(1); + }); + + it('should handle contact as JSON', async () => { + const response = await request(app) + .put('/api/external-nodes/stripe-api') + .send({ + displayName: 'Stripe API', + contact: { email: 'ops@example.com', slack: '#payments' }, + }); + + expect(response.status).toBe(200); + expect(response.body.contact).toBeTruthy(); + }); + }); + + describe('DELETE /api/external-nodes/:canonicalName', () => { + it('should delete existing enrichment', async () => { + // Create enrichment + await request(app) + .put('/api/external-nodes/stripe-api') + .send({ displayName: 'Stripe API' }); + + const response = await request(app) + .delete('/api/external-nodes/stripe-api'); + + expect(response.status).toBe(204); + + // Verify deletion + const listResponse = await request(app).get('/api/external-nodes'); + expect(listResponse.body).toHaveLength(0); + }); + + it('should return 404 for non-existent enrichment', async () => { + const response = await request(app) + .delete('/api/external-nodes/non-existent'); + + expect(response.status).toBe(404); + expect(response.body.error).toContain('not found'); + }); + }); +}); diff --git a/server/src/routes/externalNodes/index.ts b/server/src/routes/externalNodes/index.ts new file mode 100644 index 0000000..4fc9b28 --- /dev/null +++ b/server/src/routes/externalNodes/index.ts @@ -0,0 +1,61 @@ +import { Router, Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; + +const router = Router(); + +// GET /api/external-nodes - List all external node enrichment records +router.get('/', (req: Request, res: Response) => { + try { + const stores = getStores(); + const enrichments = stores.externalNodeEnrichment.findAll(); + res.status(200).json(enrichments); + } catch (error) /* istanbul ignore next */ { + sendErrorResponse(res, error, 'listing external node enrichments'); + } +}); + +// PUT /api/external-nodes/:canonicalName - Upsert enrichment for an external node +router.put('/:canonicalName', (req: Request, res: Response) => { + try { + const { canonicalName } = req.params; + const stores = getStores(); + + const { displayName, description, impact, contact, serviceType } = req.body; + + const enrichment = stores.externalNodeEnrichment.upsert({ + canonical_name: canonicalName, + display_name: displayName ?? null, + description: description ?? null, + impact: impact ?? null, + contact: contact ? JSON.stringify(contact) : null, + service_type: serviceType ?? null, + updated_by: req.user?.id ?? null, + }); + + res.status(200).json(enrichment); + } catch (error) /* istanbul ignore next */ { + sendErrorResponse(res, error, 'upserting external node enrichment'); + } +}); + +// DELETE /api/external-nodes/:canonicalName - Remove enrichment for an external node +router.delete('/:canonicalName', (req: Request, res: Response) => { + try { + const { canonicalName } = req.params; + const stores = getStores(); + + const existing = stores.externalNodeEnrichment.findByCanonicalName(canonicalName); + if (!existing) { + res.status(404).json({ error: 'External node enrichment not found' }); + return; + } + + stores.externalNodeEnrichment.delete(existing.id); + res.status(204).send(); + } catch (error) /* istanbul ignore next */ { + sendErrorResponse(res, error, 'deleting external node enrichment'); + } +}); + +export default router; diff --git a/server/src/routes/formatters/dependencyFormatter.test.ts b/server/src/routes/formatters/dependencyFormatter.test.ts index 6a9b876..f15e325 100644 --- a/server/src/routes/formatters/dependencyFormatter.test.ts +++ b/server/src/routes/formatters/dependencyFormatter.test.ts @@ -25,6 +25,7 @@ describe('dependencyFormatter', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2024-01-01T00:00:00.000Z', updated_at: '2024-01-01T00:00:00.000Z', }; @@ -34,6 +35,8 @@ describe('dependencyFormatter', () => { dependency_id: 'dep-1', linked_service_id: 'service-1', association_type: 'api_call', + is_auto_suggested: 0, + is_dismissed: 0, manifest_managed: 0, created_at: '2024-01-01T00:00:00.000Z', }; @@ -56,6 +59,10 @@ describe('dependencyFormatter', () => { check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: '2024-01-01T00:00:00.000Z', last_status_change: null, diff --git a/server/src/routes/formatters/serviceFormatter.test.ts b/server/src/routes/formatters/serviceFormatter.test.ts index c47ba2a..ca0efa8 100644 --- a/server/src/routes/formatters/serviceFormatter.test.ts +++ b/server/src/routes/formatters/serviceFormatter.test.ts @@ -46,6 +46,7 @@ describe('serviceFormatter', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2024-01-01T00:00:00.000Z', updated_at: '2024-01-01T00:00:00.000Z', team_name: 'Test Team', @@ -76,6 +77,10 @@ describe('serviceFormatter', () => { check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: '2024-01-01T00:00:00.000Z', last_status_change: null, @@ -323,6 +328,7 @@ describe('serviceFormatter', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2024-01-01T00:00:00.000Z', updated_at: '2024-01-01T00:00:00.000Z', }; @@ -368,6 +374,7 @@ describe('serviceFormatter', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2024-01-01T00:00:00.000Z', updated_at: '2024-01-01T00:00:00.000Z', }; diff --git a/server/src/routes/formatters/serviceFormatter.ts b/server/src/routes/formatters/serviceFormatter.ts index 8eda01f..be3312c 100644 --- a/server/src/routes/formatters/serviceFormatter.ts +++ b/server/src/routes/formatters/serviceFormatter.ts @@ -40,6 +40,7 @@ function extractServiceFields(row: Service | ServiceWithTeam) { description: row.description ?? null, last_poll_success: row.last_poll_success ?? null, last_poll_error: row.last_poll_error ?? null, + health_endpoint_format: row.health_endpoint_format ?? 'default', poll_warnings: row.poll_warnings ?? null, manifest_managed: row.manifest_managed ?? 0, manifest_key: row.manifest_key ?? null, diff --git a/server/src/routes/formatters/types.ts b/server/src/routes/formatters/types.ts index 3267648..8614d4c 100644 --- a/server/src/routes/formatters/types.ts +++ b/server/src/routes/formatters/types.ts @@ -1,4 +1,4 @@ -import { AggregatedHealth, Team, Service, Dependency, DependentReport } from '../../db/types'; +import { AggregatedHealth, Team, Service, Dependency, DependentReport, HealthEndpointFormat } from '../../db/types'; import { DependencyWithResolvedOverrides } from '../../stores/types'; // Formatted team embedded in service response @@ -29,6 +29,7 @@ export interface FormattedServiceListItem { is_active: number; last_poll_success: number | null; last_poll_error: string | null; + health_endpoint_format: HealthEndpointFormat; poll_warnings: string | null; manifest_managed: number; manifest_key: string | null; @@ -55,6 +56,7 @@ export interface FormattedServiceMutation { is_active: number; last_poll_success: number | null; last_poll_error: string | null; + health_endpoint_format: HealthEndpointFormat; poll_warnings: string | null; manifest_managed: number; manifest_key: string | null; diff --git a/server/src/routes/graph/graph.test.ts b/server/src/routes/graph/graph.test.ts index 00991e1..98873cf 100644 --- a/server/src/routes/graph/graph.test.ts +++ b/server/src/routes/graph/graph.test.ts @@ -46,6 +46,11 @@ describe('Graph API', () => { last_poll_success INTEGER, last_poll_error TEXT, poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')) ); @@ -68,6 +73,10 @@ describe('Graph API', () => { check_details TEXT, error TEXT, error_message TEXT, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, skipped INTEGER NOT NULL DEFAULT 0, last_checked TEXT, last_status_change TEXT, @@ -81,6 +90,8 @@ describe('Graph API', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), UNIQUE (dependency_id, linked_service_id) @@ -96,8 +107,23 @@ describe('Graph API', () => { CREATE TABLE dependency_canonical_overrides ( id TEXT PRIMARY KEY, canonical_name TEXT NOT NULL UNIQUE, + team_id TEXT, contact_override TEXT, impact_override TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT + ); + + CREATE TABLE external_node_enrichment ( + id TEXT PRIMARY KEY, + canonical_name TEXT NOT NULL UNIQUE, + display_name TEXT, + description TEXT, + impact TEXT, + contact TEXT, + service_type TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')), updated_by TEXT diff --git a/server/src/routes/latency/aggregateLatencyBuckets.test.ts b/server/src/routes/latency/aggregateLatencyBuckets.test.ts index 2d91a11..c2c288f 100644 --- a/server/src/routes/latency/aggregateLatencyBuckets.test.ts +++ b/server/src/routes/latency/aggregateLatencyBuckets.test.ts @@ -37,6 +37,13 @@ describe('Aggregate Latency Buckets API', () => { id TEXT PRIMARY KEY, dependency_id TEXT NOT NULL, latency_ms INTEGER NOT NULL, + p50_ms REAL, + p95_ms REAL, + p99_ms REAL, + min_ms REAL, + max_ms REAL, + request_count INTEGER, + source TEXT NOT NULL DEFAULT 'poll', recorded_at TEXT NOT NULL ); diff --git a/server/src/routes/latency/latency.test.ts b/server/src/routes/latency/latency.test.ts index fb3bacb..45a03ae 100644 --- a/server/src/routes/latency/latency.test.ts +++ b/server/src/routes/latency/latency.test.ts @@ -36,6 +36,13 @@ describe('Latency API', () => { id TEXT PRIMARY KEY, dependency_id TEXT NOT NULL, latency_ms INTEGER NOT NULL, + p50_ms REAL, + p95_ms REAL, + p99_ms REAL, + min_ms REAL, + max_ms REAL, + request_count INTEGER, + source TEXT NOT NULL DEFAULT 'poll', recorded_at TEXT NOT NULL ); diff --git a/server/src/routes/latency/latencyBuckets.test.ts b/server/src/routes/latency/latencyBuckets.test.ts index 690200c..c522e8b 100644 --- a/server/src/routes/latency/latencyBuckets.test.ts +++ b/server/src/routes/latency/latencyBuckets.test.ts @@ -36,6 +36,13 @@ describe('Latency Buckets API', () => { id TEXT PRIMARY KEY, dependency_id TEXT NOT NULL, latency_ms INTEGER NOT NULL, + p50_ms REAL, + p95_ms REAL, + p99_ms REAL, + min_ms REAL, + max_ms REAL, + request_count INTEGER, + source TEXT NOT NULL DEFAULT 'poll', recorded_at TEXT NOT NULL ); diff --git a/server/src/routes/otlp/index.test.ts b/server/src/routes/otlp/index.test.ts new file mode 100644 index 0000000..65cdd55 --- /dev/null +++ b/server/src/routes/otlp/index.test.ts @@ -0,0 +1,710 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +// Mock requireApiKeyAuth to inject apiKeyTeamId and apiKeyId +const MOCK_TEAM_ID = 'team-1'; +const MOCK_API_KEY_ID = 'api-key-1'; +let mockApiKeyTeamId: string | undefined = MOCK_TEAM_ID; +let mockApiKeyId: string | undefined = MOCK_API_KEY_ID; + +jest.mock('../../auth/apiKeyAuth', () => ({ + requireApiKeyAuth: jest.fn((req: Record, _res: unknown, next: () => void) => { + if (mockApiKeyTeamId) { + req.apiKeyTeamId = mockApiKeyTeamId; + req.apiKeyId = mockApiKeyId; + next(); + } else { + const res = _res as { status: (code: number) => { json: (body: unknown) => void } }; + res.status(401).json({ error: 'Invalid API key' }); + } + }), +})); + +// Mock HealthPollingService to avoid singleton issues in tests +const mockEmit = jest.fn(); +jest.mock('../../services/polling', () => ({ + HealthPollingService: { + getInstance: jest.fn(() => ({ + emit: mockEmit, + })), + }, + PollingEventType: { + STATUS_CHANGE: 'status:change', + POLL_COMPLETE: 'poll:complete', + POLL_ERROR: 'poll:error', + SERVICE_STARTED: 'service:started', + SERVICE_STOPPED: 'service:stopped', + CIRCUIT_OPEN: 'circuit:open', + CIRCUIT_CLOSE: 'circuit:close', + }, +})); + +import { requireApiKeyAuth } from '../../auth/apiKeyAuth'; +import { createPerKeyRateLimit, evictBucket } from '../../middleware/perKeyRateLimit'; +import { createTrackApiKeyUsage, _accumulator } from '../../middleware/trackApiKeyUsage'; +import otlpRouter from './index'; + +const app = express(); +app.use(express.json({ limit: '1mb' })); +app.use('/v1/metrics', requireApiKeyAuth, otlpRouter); + +function buildOtlpPayload( + serviceName: string, + dependencies: Array<{ + name: string; + status?: number; + healthy?: number; + latency?: number; + code?: number; + type?: string; + }>, +) { + return { + resourceMetrics: [ + { + resource: { + attributes: [ + { key: 'service.name', value: { stringValue: serviceName } }, + ], + }, + scopeMetrics: [ + { + metrics: dependencies.flatMap((dep) => { + const metrics = []; + const attrs = [ + { key: 'dependency.name', value: { stringValue: dep.name } }, + ]; + if (dep.type) { + attrs.push({ key: 'dependency.type', value: { stringValue: dep.type } }); + } + + if (dep.status !== undefined) { + metrics.push({ + name: 'dependency.health.status', + gauge: { + dataPoints: [{ asInt: String(dep.status), attributes: attrs, timeUnixNano: '1700000000000000000' }], + }, + }); + } + if (dep.healthy !== undefined) { + metrics.push({ + name: 'dependency.health.healthy', + gauge: { + dataPoints: [{ asInt: String(dep.healthy), attributes: attrs, timeUnixNano: '1700000000000000000' }], + }, + }); + } + if (dep.latency !== undefined) { + metrics.push({ + name: 'dependency.health.latency', + gauge: { + dataPoints: [{ asDouble: dep.latency, attributes: attrs, timeUnixNano: '1700000000000000000' }], + }, + }); + } + if (dep.code !== undefined) { + metrics.push({ + name: 'dependency.health.code', + gauge: { + dataPoints: [{ asInt: String(dep.code), attributes: attrs, timeUnixNano: '1700000000000000000' }], + }, + }); + } + return metrics; + }), + }, + ], + }, + ], + }; +} + +afterAll(() => { + testDb.close(); +}); + +describe('OTLP Receiver Route', () => { + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE RESTRICT + ); + CREATE INDEX idx_services_team_id ON services(team_id); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms INTEGER, + last_checked TEXT, + last_status_change TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + contact TEXT, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE, + UNIQUE (service_id, name) + ); + CREATE INDEX idx_dependencies_service_id ON dependencies(service_id); + + CREATE TABLE dependency_latency_history ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + latency_ms INTEGER NOT NULL, + recorded_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE + ); + + CREATE TABLE dependency_aliases ( + id TEXT PRIMARY KEY, + alias TEXT NOT NULL UNIQUE, + canonical_name TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE dependency_error_history ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + error TEXT, + error_message TEXT, + recorded_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + `); + + // Insert test team + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(MOCK_TEAM_ID, 'Test Team', 'TEST'); + }); + + beforeEach(() => { + mockApiKeyTeamId = MOCK_TEAM_ID; + mockApiKeyId = MOCK_API_KEY_ID; + mockEmit.mockClear(); + // Clean up services and dependencies between tests + testDb.exec('DELETE FROM dependency_latency_history'); + testDb.exec('DELETE FROM dependency_error_history'); + testDb.exec('DELETE FROM dependencies'); + testDb.exec('DELETE FROM services'); + }); + + describe('POST /v1/metrics', () => { + it('should accept a valid OTLP payload and return success', async () => { + const payload = buildOtlpPayload('my-service', [ + { name: 'postgres', status: 0, healthy: 1, latency: 5, code: 200, type: 'database' }, + ]); + + const res = await request(app) + .post('/v1/metrics') + .send(payload); + + expect(res.status).toBe(200); + expect(res.body.partialSuccess.rejectedDataPoints).toBe(0); + expect(res.body.partialSuccess.errorMessage).toBe(''); + }); + + it('should auto-register an unknown service with otlp format', async () => { + const payload = buildOtlpPayload('auto-registered-svc', [ + { name: 'redis', status: 0, healthy: 1 }, + ]); + + await request(app) + .post('/v1/metrics') + .send(payload); + + const service = testDb + .prepare('SELECT * FROM services WHERE name = ? AND team_id = ?') + .get('auto-registered-svc', MOCK_TEAM_ID) as Record | undefined; + + expect(service).toBeDefined(); + expect(service!.health_endpoint_format).toBe('otlp'); + expect(service!.health_endpoint).toBe(''); + expect(service!.poll_interval_ms).toBe(0); + expect(service!.is_active).toBe(1); + }); + + it('should upsert dependencies for auto-registered service', async () => { + const payload = buildOtlpPayload('dep-test-svc', [ + { name: 'postgres', status: 0, healthy: 1, latency: 12, code: 200 }, + { name: 'redis', status: 0, healthy: 1, latency: 3, code: 200 }, + ]); + + await request(app) + .post('/v1/metrics') + .send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('dep-test-svc') as Array>; + + expect(deps).toHaveLength(2); + const names = deps.map((d) => d.name); + expect(names).toContain('postgres'); + expect(names).toContain('redis'); + }); + + it('should handle idempotent push (second push updates, not duplicates)', async () => { + const payload = buildOtlpPayload('idempotent-svc', [ + { name: 'postgres', status: 0, healthy: 1, latency: 5 }, + ]); + + await request(app).post('/v1/metrics').send(payload); + await request(app).post('/v1/metrics').send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('idempotent-svc') as Array>; + + expect(deps).toHaveLength(1); + }); + + it('should use existing service and warn if format is not otlp', async () => { + // Pre-create a service with 'default' format + testDb.prepare(` + INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, poll_interval_ms) + VALUES (?, ?, ?, ?, ?, ?) + `).run('existing-svc-id', 'existing-svc', MOCK_TEAM_ID, 'http://localhost:8080/health', 'default', 30000); + + const payload = buildOtlpPayload('existing-svc', [ + { name: 'postgres', status: 0, healthy: 1 }, + ]); + + const res = await request(app) + .post('/v1/metrics') + .send(payload); + + expect(res.status).toBe(200); + expect(res.body.partialSuccess.errorMessage).toContain('exists with format "default"'); + // Should NOT overwrite the format + const service = testDb.prepare('SELECT * FROM services WHERE id = ?').get('existing-svc-id') as Record; + expect(service.health_endpoint_format).toBe('default'); + }); + + it('should return 400 for invalid OTLP payload', async () => { + const res = await request(app) + .post('/v1/metrics') + .send({ invalid: 'data' }); + + expect(res.status).toBe(400); + expect(res.body.partialSuccess.errorMessage).toBeTruthy(); + }); + + it('should return 400 for missing resourceMetrics', async () => { + const res = await request(app) + .post('/v1/metrics') + .send({ resourceMetrics: 'not-an-array' }); + + expect(res.status).toBe(400); + }); + + it('should handle payload with multiple services', async () => { + const payload = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc-a' } }], + }, + scopeMetrics: [{ + metrics: [{ + name: 'dependency.health.status', + gauge: { + dataPoints: [{ + asInt: '0', + attributes: [{ key: 'dependency.name', value: { stringValue: 'dep-a' } }], + timeUnixNano: '1700000000000000000', + }], + }, + }], + }], + }, + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc-b' } }], + }, + scopeMetrics: [{ + metrics: [{ + name: 'dependency.health.status', + gauge: { + dataPoints: [{ + asInt: '0', + attributes: [{ key: 'dependency.name', value: { stringValue: 'dep-b' } }], + timeUnixNano: '1700000000000000000', + }], + }, + }], + }], + }, + ], + }; + + const res = await request(app) + .post('/v1/metrics') + .send(payload); + + expect(res.status).toBe(200); + + const services = testDb + .prepare('SELECT * FROM services WHERE team_id = ?') + .all(MOCK_TEAM_ID) as Array>; + + expect(services).toHaveLength(2); + const names = services.map((s) => s.name); + expect(names).toContain('svc-a'); + expect(names).toContain('svc-b'); + }); + + it('should handle empty dependencies gracefully', async () => { + const payload = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'empty-svc' } }], + }, + scopeMetrics: [], + }, + ], + }; + + const res = await request(app) + .post('/v1/metrics') + .send(payload); + + expect(res.status).toBe(200); + expect(res.body.partialSuccess.rejectedDataPoints).toBe(0); + }); + + it('should not create service in a different team', async () => { + // Create another team + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run('team-2', 'Other Team', 'OTHER'); + + // Push with team-1 API key + const payload = buildOtlpPayload('team1-svc', [ + { name: 'dep1', status: 0, healthy: 1 }, + ]); + + await request(app).post('/v1/metrics').send(payload); + + // Verify service belongs to team-1 + const service = testDb + .prepare('SELECT * FROM services WHERE name = ?') + .get('team1-svc') as Record; + expect(service.team_id).toBe(MOCK_TEAM_ID); + + // Clean up + testDb.exec("DELETE FROM teams WHERE id = 'team-2'"); + }); + + it('should report rejected data points on partial failure', async () => { + // Create a payload with missing service.name — handled gracefully with warning + const payload = { + resourceMetrics: [ + { + resource: { attributes: [] }, + scopeMetrics: [{ + metrics: [{ + name: 'dependency.health.status', + gauge: { + dataPoints: [{ + asInt: '0', + attributes: [{ key: 'dependency.name', value: { stringValue: 'dep1' } }], + }], + }, + }], + }], + }, + ], + }; + + const res = await request(app) + .post('/v1/metrics') + .send(payload); + + // Should return 200 with warning about missing service.name + expect(res.status).toBe(200); + expect(res.body.partialSuccess.errorMessage).toContain('missing service.name'); + }); + }); +}); + +describe('OTLP Receiver — Per-Key Rate Limiting', () => { + // Use a fixed time so token bucket behavior is deterministic + let testNow = 1700000000000; + const getNow = () => testNow; + + // Separate app with the full middleware chain including per-key rate limit and usage tracking + const rateLimitedApp = express(); + rateLimitedApp.use(express.json({ limit: '1mb' })); + rateLimitedApp.use( + '/v1/metrics', + requireApiKeyAuth, + createPerKeyRateLimit({ getNow }), + createTrackApiKeyUsage(), + otlpRouter, + ); + + function buildSimplePayload() { + return { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'rate-limit-test-svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { + dataPoints: [ + { + asInt: '0', + attributes: [{ key: 'dependency.name', value: { stringValue: 'dep-rl' } }], + timeUnixNano: '1700000000000000000', + }, + ], + }, + }, + ], + }, + ], + }, + ], + }; + } + + beforeEach(() => { + mockApiKeyTeamId = MOCK_TEAM_ID; + mockApiKeyId = MOCK_API_KEY_ID; + testNow = 1700000000000; + mockEmit.mockClear(); + _accumulator.clear(); + evictBucket(MOCK_API_KEY_ID); + + // Clean up test data + testDb.exec('DELETE FROM api_key_usage_buckets'); + testDb.exec('DELETE FROM team_api_keys'); + testDb.exec('DELETE FROM dependencies'); + testDb.exec('DELETE FROM services'); + }); + + it('should return 200 with RateLimit headers when within limit', async () => { + // Insert an API key with a reasonable rate limit + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)', + ).run(MOCK_API_KEY_ID, MOCK_TEAM_ID, 'Test Key', 'hash-test', 'dps_test', 600, '2026-01-01T00:00:00Z'); + + const res = await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + + expect(res.status).toBe(200); + expect(res.headers['ratelimit-limit']).toBe('600'); + expect(res.headers['ratelimit-remaining']).toBeDefined(); + expect(res.headers['ratelimit-reset']).toBeDefined(); + expect(res.headers['x-ratelimit-key']).toBe('dps_test'); + expect(res.headers['retry-after']).toBeUndefined(); + }); + + it('should return 429 with partialSuccess body when rate limit is exceeded', async () => { + // Insert a key with very low limit: 60 rpm → burst = ceil(60/60 * 6) = 6 tokens + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)', + ).run(MOCK_API_KEY_ID, MOCK_TEAM_ID, 'Low Limit Key', 'hash-low', 'dps_low', 60, '2026-01-01T00:00:00Z'); + + // Exhaust all 6 tokens (time frozen, no refill) + for (let i = 0; i < 6; i++) { + const res = await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + expect(res.status).toBe(200); + } + + // 7th request should be rejected + const rejectedRes = await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + + expect(rejectedRes.status).toBe(429); + expect(rejectedRes.body.partialSuccess).toBeDefined(); + expect(rejectedRes.body.partialSuccess.rejectedDataPoints).toBe(0); + expect(rejectedRes.body.partialSuccess.errorMessage).toContain('Rate limit exceeded'); + expect(rejectedRes.headers['retry-after']).toBeDefined(); + expect(rejectedRes.headers['ratelimit-remaining']).toBe('0'); + }); + + it('should increment rejected_count but not push_count for rejected requests', async () => { + // 60 rpm → 6 token burst capacity + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)', + ).run(MOCK_API_KEY_ID, MOCK_TEAM_ID, 'Track Key', 'hash-track', 'dps_trk', 60, '2026-01-01T00:00:00Z'); + + // Send 6 allowed requests + for (let i = 0; i < 6; i++) { + await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + } + + // Send 2 rejected requests + await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + + // Check the in-memory accumulator for correctness + let totalCount = 0; + let totalRejected = 0; + for (const [, entry] of _accumulator) { + totalCount += entry.count; + totalRejected += entry.rejected; + } + + // 6 allowed requests tracked across 2 granularities = 12 total count entries + // But count per-granularity should be 6 + // 2 rejected requests across 2 granularities = 4 total rejected entries + // But rejected per-granularity should be 2 + // Total count across all entries: each allowed request increments count in 2 entries (minute+hour) + expect(totalCount).toBe(12); // 6 requests × 2 granularities + expect(totalRejected).toBe(4); // 2 rejections × 2 granularities + }); + + it('should not rate limit when key has rate_limit_rpm = 0 (unlimited)', async () => { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)', + ).run(MOCK_API_KEY_ID, MOCK_TEAM_ID, 'Unlimited Key', 'hash-unlim', 'dps_unlm', 0, '2026-01-01T00:00:00Z'); + + // Send many requests — all should pass + for (let i = 0; i < 20; i++) { + const res = await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + expect(res.status).toBe(200); + } + }); + + it('should include X-RateLimit-Warning header when near limit', async () => { + // 60 rpm → 6 token capacity. 80% threshold means warning at 5+ consumed (1 remaining) + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)', + ).run(MOCK_API_KEY_ID, MOCK_TEAM_ID, 'Warn Key', 'hash-warn', 'dps_warn', 60, '2026-01-01T00:00:00Z'); + + // First request should NOT have warning (1/6 consumed = 17%) + const firstRes = await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + expect(firstRes.status).toBe(200); + expect(firstRes.headers['x-ratelimit-warning']).toBeUndefined(); + + // Consume tokens until we hit the warning threshold + for (let i = 0; i < 4; i++) { + await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + } + + // 6th request: 5 consumed = 83% → should have warning + const warnRes = await request(rateLimitedApp) + .post('/v1/metrics') + .send(buildSimplePayload()); + expect(warnRes.status).toBe(200); + expect(warnRes.headers['x-ratelimit-warning']).toBe('true'); + }); +}); diff --git a/server/src/routes/otlp/index.ts b/server/src/routes/otlp/index.ts new file mode 100644 index 0000000..fb0b8b6 --- /dev/null +++ b/server/src/routes/otlp/index.ts @@ -0,0 +1,115 @@ +import { Router, Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { OtlpParser } from '../../services/polling/OtlpParser'; +import { getDependencyUpsertService } from '../../services/polling/DependencyUpsertService'; +import { HealthPollingService } from '../../services/polling'; +import { Service, MetricSchemaConfig } from '../../db/types'; +import { isMetricSchemaConfig } from '../../services/polling/metricSchemaUtils'; +import { StatusChangeEvent, PollingEventType } from '../../services/polling/types'; +import { OtlpExportMetricsServiceRequest } from '../../services/polling/otlp-types'; +import { findOrCreateService } from '../../services/polling/otlpServiceResolver'; +import logger from '../../utils/logger'; + +const router = Router(); +const parser = new OtlpParser(); + +/** + * POST /v1/metrics + * OTLP JSON metrics receiver. Authenticated via API key (requireApiKeyAuth middleware). + * Parses OTLP payload per-service with config-aware metric mapping. + */ +router.post('/', (req: Request, res: Response): void => { + const teamId = req.apiKeyTeamId; + + if (!teamId) { + res.status(401).json({ error: 'Missing team context' }); + return; + } + + // Validate basic OTLP structure + const data = req.body; + if (!data || typeof data !== 'object' || !Array.isArray(data.resourceMetrics)) { + logger.warn('OTLP parse error: invalid payload structure'); + res.status(400).json({ + partialSuccess: { + rejectedDataPoints: -1, + errorMessage: 'Invalid OTLP payload: expected object with resourceMetrics array', + }, + }); + return; + } + + const request = data as OtlpExportMetricsServiceRequest; + const stores = getStores(); + const upsertService = getDependencyUpsertService(); + const warnings: string[] = []; + let totalRejected = 0; + const allChanges: StatusChangeEvent[] = []; + + // Process each resourceMetrics entry with per-service config + for (const rm of request.resourceMetrics) { + try { + const serviceName = parser.extractServiceName(rm); + if (!serviceName) { + warnings.push('Skipping resourceMetrics entry: missing service.name resource attribute'); + continue; + } + + // Find or auto-register the service + const service = findOrCreateService(stores, teamId, serviceName, warnings); + + // Load per-service metric schema config + const metricConfig = loadMetricConfig(service); + + // Parse this resourceMetrics with the service's config + const result = parser.parseResourceMetrics(rm, metricConfig); + warnings.push(...parser.lastWarnings); + + // Upsert dependencies + const changes = upsertService.upsert(service, result.dependencies); + allChanges.push(...changes); + + // Update poll result on the service + stores.services.updatePollResult(service.id, true, undefined, warnings.length > 0 ? warnings : undefined); + } catch (err) { + const message = err instanceof Error ? err.message : 'Unknown error'; + logger.error({ err }, 'OTLP processing failed for resourceMetrics entry'); + warnings.push(message); + totalRejected++; + } + } + + // Emit status change events to the polling service for alert processing + if (allChanges.length > 0) { + try { + const pollingService = HealthPollingService.getInstance(); + for (const change of allChanges) { + pollingService.emit(PollingEventType.STATUS_CHANGE, change); + } + } catch { + // Polling service may not be initialized in tests — non-critical + } + } + + res.status(200).json({ + partialSuccess: { + rejectedDataPoints: totalRejected, + errorMessage: warnings.length > 0 ? warnings.join('; ') : '', + }, + }); +}); + +/** + * Load a MetricSchemaConfig from a service's schema_config if present and valid. + */ +function loadMetricConfig(service: Service): MetricSchemaConfig | undefined { + if (!service.schema_config) return undefined; + try { + const parsed = JSON.parse(service.schema_config); + return isMetricSchemaConfig(parsed) ? parsed : undefined; + } catch { + return undefined; + } +} + +export default router; diff --git a/server/src/routes/otlp/traces.test.ts b/server/src/routes/otlp/traces.test.ts new file mode 100644 index 0000000..b340646 --- /dev/null +++ b/server/src/routes/otlp/traces.test.ts @@ -0,0 +1,787 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +// Mock requireApiKeyAuth to inject apiKeyTeamId and apiKeyId +const MOCK_TEAM_ID = 'team-1'; +const MOCK_API_KEY_ID = 'api-key-1'; +let mockApiKeyTeamId: string | undefined = MOCK_TEAM_ID; + +jest.mock('../../auth/apiKeyAuth', () => ({ + requireApiKeyAuth: jest.fn((req: Record, _res: unknown, next: () => void) => { + if (mockApiKeyTeamId) { + req.apiKeyTeamId = mockApiKeyTeamId; + req.apiKeyId = MOCK_API_KEY_ID; + next(); + } else { + const res = _res as { status: (code: number) => { json: (body: unknown) => void } }; + res.status(401).json({ error: 'Invalid API key' }); + } + }), +})); + +// Mock HealthPollingService to avoid singleton issues in tests +const mockEmit = jest.fn(); +jest.mock('../../services/polling', () => ({ + HealthPollingService: { + getInstance: jest.fn(() => ({ + emit: mockEmit, + })), + }, + PollingEventType: { + STATUS_CHANGE: 'status:change', + POLL_COMPLETE: 'poll:complete', + POLL_ERROR: 'poll:error', + SERVICE_STARTED: 'service:started', + SERVICE_STOPPED: 'service:stopped', + CIRCUIT_OPEN: 'circuit:open', + CIRCUIT_CLOSE: 'circuit:close', + }, +})); + +import { requireApiKeyAuth } from '../../auth/apiKeyAuth'; +import traceRouter from './traces'; + +const app = express(); +app.use(express.json({ limit: '2mb' })); +app.use('/v1/traces', requireApiKeyAuth, traceRouter); + +/** + * Build an OTLP trace payload with the given spans. + */ +function buildTracePayload( + serviceName: string, + spans: Array<{ + traceId?: string; + spanId?: string; + parentSpanId?: string; + name: string; + kind?: number; // 1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER + startTimeUnixNano?: string; + endTimeUnixNano?: string; + attributes?: Array<{ key: string; value: { stringValue?: string; intValue?: string } }>; + status?: { code?: number; message?: string }; + }>, +) { + return { + resourceSpans: [ + { + resource: { + attributes: [ + { key: 'service.name', value: { stringValue: serviceName } }, + ], + }, + scopeSpans: [ + { + scope: { name: 'test' }, + spans: spans.map((s, i) => ({ + traceId: s.traceId ?? 'trace-001', + spanId: s.spanId ?? `span-${i}`, + parentSpanId: s.parentSpanId, + name: s.name, + kind: s.kind ?? 3, // default CLIENT + startTimeUnixNano: s.startTimeUnixNano ?? '1700000000000000000', + endTimeUnixNano: s.endTimeUnixNano ?? '1700000050000000000', // 50ms later + attributes: s.attributes ?? [ + { key: 'peer.service', value: { stringValue: 'target-svc' } }, + ], + status: s.status, + })), + }, + ], + }, + ], + }; +} + +afterAll(() => { + testDb.close(); +}); + +describe('OTLP Trace Receiver Route', () => { + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE RESTRICT + ); + CREATE INDEX idx_services_team_id ON services(team_id); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms INTEGER, + last_checked TEXT, + last_status_change TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + contact TEXT, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE, + UNIQUE (service_id, name) + ); + CREATE INDEX idx_dependencies_service_id ON dependencies(service_id); + + CREATE TABLE dependency_latency_history ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + latency_ms INTEGER NOT NULL, + recorded_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE + ); + + CREATE TABLE dependency_aliases ( + id TEXT PRIMARY KEY, + alias TEXT NOT NULL UNIQUE, + canonical_name TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE dependency_error_history ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + error TEXT, + error_message TEXT, + recorded_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE + ); + + CREATE TABLE spans ( + id TEXT PRIMARY KEY, + trace_id TEXT NOT NULL, + span_id TEXT NOT NULL, + parent_span_id TEXT, + service_name TEXT NOT NULL, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + kind INTEGER NOT NULL DEFAULT 0, + start_time TEXT NOT NULL, + end_time TEXT NOT NULL, + duration_ms REAL NOT NULL, + status_code INTEGER DEFAULT 0, + status_message TEXT, + attributes TEXT, + resource_attributes TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE INDEX idx_spans_trace_id ON spans(trace_id); + CREATE INDEX idx_spans_service_team ON spans(service_name, team_id); + CREATE INDEX idx_spans_start_time ON spans(start_time); + + CREATE TABLE dependency_associations ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + linked_service_id TEXT NOT NULL, + association_type TEXT NOT NULL DEFAULT 'other', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, + manifest_managed INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE, + FOREIGN KEY (linked_service_id) REFERENCES services(id) ON DELETE CASCADE, + UNIQUE (dependency_id, linked_service_id) + ); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + `); + + // Insert test team + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(MOCK_TEAM_ID, 'Test Team', 'TEST'); + }); + + beforeEach(() => { + mockApiKeyTeamId = MOCK_TEAM_ID; + mockEmit.mockClear(); + // Clean up between tests + testDb.exec('DELETE FROM dependency_associations'); + testDb.exec('DELETE FROM spans'); + testDb.exec('DELETE FROM dependency_latency_history'); + testDb.exec('DELETE FROM dependency_error_history'); + testDb.exec('DELETE FROM dependencies'); + testDb.exec('DELETE FROM services'); + }); + + describe('POST /v1/traces', () => { + it('should accept a valid trace payload and return success', async () => { + const payload = buildTracePayload('my-service', [ + { name: 'GET /api/users', kind: 3, attributes: [{ key: 'peer.service', value: { stringValue: 'user-db' } }] }, + ]); + + const res = await request(app) + .post('/v1/traces') + .send(payload); + + expect(res.status).toBe(200); + expect(res.body.partialSuccess.rejectedDataPoints).toBe(0); + expect(res.body.partialSuccess.errorMessage).toBe(''); + }); + + it('should auto-register an unknown service with otlp format', async () => { + const payload = buildTracePayload('auto-trace-svc', [ + { name: 'db-query', kind: 3, attributes: [{ key: 'db.system', value: { stringValue: 'postgresql' } }] }, + ]); + + await request(app) + .post('/v1/traces') + .send(payload); + + const service = testDb + .prepare('SELECT * FROM services WHERE name = ? AND team_id = ?') + .get('auto-trace-svc', MOCK_TEAM_ID) as Record | undefined; + + expect(service).toBeDefined(); + expect(service!.health_endpoint_format).toBe('otlp'); + expect(service!.health_endpoint).toBe(''); + expect(service!.poll_interval_ms).toBe(0); + }); + + it('should create dependencies from CLIENT spans', async () => { + const payload = buildTracePayload('dep-trace-svc', [ + { + name: 'SELECT users', + kind: 3, + attributes: [ + { key: 'peer.service', value: { stringValue: 'postgres' } }, + { key: 'db.system', value: { stringValue: 'postgresql' } }, + ], + }, + { + name: 'GET /api/data', + spanId: 'span-2', + kind: 3, + attributes: [ + { key: 'peer.service', value: { stringValue: 'data-service' } }, + { key: 'http.request.method', value: { stringValue: 'GET' } }, + ], + }, + ]); + + await request(app) + .post('/v1/traces') + .send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('dep-trace-svc') as Array>; + + expect(deps).toHaveLength(2); + const names = deps.map((d) => d.name); + expect(names).toContain('postgres'); + expect(names).toContain('data-service'); + }); + + it('should store ALL spans, not just CLIENT spans', async () => { + const payload = buildTracePayload('all-spans-svc', [ + { + name: 'client-call', + spanId: 'span-client', + kind: 3, // CLIENT + attributes: [{ key: 'peer.service', value: { stringValue: 'target' } }], + }, + { + name: 'server-handler', + spanId: 'span-server', + kind: 2, // SERVER + attributes: [], + }, + { + name: 'internal-op', + spanId: 'span-internal', + kind: 1, // INTERNAL + attributes: [], + }, + { + name: 'consumer-read', + spanId: 'span-consumer', + kind: 5, // CONSUMER + attributes: [], + }, + ]); + + await request(app) + .post('/v1/traces') + .send(payload); + + const spans = testDb + .prepare('SELECT * FROM spans WHERE service_name = ?') + .all('all-spans-svc') as Array>; + + // All 4 spans should be stored + expect(spans).toHaveLength(4); + const kinds = spans.map((s) => s.kind); + expect(kinds).toContain(1); // INTERNAL + expect(kinds).toContain(2); // SERVER + expect(kinds).toContain(3); // CLIENT + expect(kinds).toContain(5); // CONSUMER + }); + + it('should create dependency when CLIENT span targets uninstrumented DB', async () => { + const payload = buildTracePayload('db-caller-svc', [ + { + name: 'SELECT * FROM orders', + kind: 3, + attributes: [ + { key: 'db.system', value: { stringValue: 'mysql' } }, + { key: 'db.operation', value: { stringValue: 'SELECT' } }, + { key: 'db.namespace', value: { stringValue: 'shop' } }, + ], + }, + ]); + + await request(app) + .post('/v1/traces') + .send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('db-caller-svc') as Array>; + + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('mysql'); + expect(deps[0].type).toBe('database'); + }); + + it('should fall back to server.address when peer.service is missing', async () => { + const payload = buildTracePayload('fallback-svc', [ + { + name: 'GET /health', + kind: 3, + attributes: [ + { key: 'server.address', value: { stringValue: 'api.example.com' } }, + { key: 'http.request.method', value: { stringValue: 'GET' } }, + ], + }, + ]); + + await request(app) + .post('/v1/traces') + .send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('fallback-svc') as Array>; + + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('api.example.com'); + expect(deps[0].type).toBe('rest'); + }); + + it('should return 400 for invalid payload structure', async () => { + const res = await request(app) + .post('/v1/traces') + .send({ invalid: 'data' }); + + expect(res.status).toBe(400); + expect(res.body.partialSuccess.errorMessage).toContain('Invalid OTLP payload'); + }); + + it('should return 400 for missing resourceSpans', async () => { + const res = await request(app) + .post('/v1/traces') + .send({ resourceSpans: 'not-an-array' }); + + expect(res.status).toBe(400); + }); + + it('should return 401 when API key auth fails', async () => { + mockApiKeyTeamId = undefined; + + const payload = buildTracePayload('unauth-svc', [ + { name: 'call', kind: 3, attributes: [{ key: 'peer.service', value: { stringValue: 'target' } }] }, + ]); + + const res = await request(app) + .post('/v1/traces') + .send(payload); + + expect(res.status).toBe(401); + }); + + it('should emit status change events on health change', async () => { + // First push: healthy + const payload1 = buildTracePayload('events-svc', [ + { + name: 'call-target', + kind: 3, + attributes: [{ key: 'peer.service', value: { stringValue: 'target-dep' } }], + status: { code: 0 }, + }, + ]); + + await request(app).post('/v1/traces').send(payload1); + + // Second push: error status + const payload2 = buildTracePayload('events-svc', [ + { + name: 'call-target', + kind: 3, + attributes: [{ key: 'peer.service', value: { stringValue: 'target-dep' } }], + status: { code: 2, message: 'connection refused' }, + }, + ]); + + await request(app).post('/v1/traces').send(payload2); + + // Should have emitted a status change event + expect(mockEmit).toHaveBeenCalled(); + const emittedEvent = mockEmit.mock.calls[0]; + expect(emittedEvent[0]).toBe('status:change'); + expect(emittedEvent[1]).toMatchObject({ + serviceName: 'events-svc', + dependencyName: 'target-dep', + previousHealthy: true, + currentHealthy: false, + }); + }); + + it('should handle idempotent push (second push updates, not duplicates)', async () => { + const payload = buildTracePayload('idem-trace-svc', [ + { + name: 'call', + kind: 3, + attributes: [{ key: 'peer.service', value: { stringValue: 'target' } }], + }, + ]); + + await request(app).post('/v1/traces').send(payload); + await request(app).post('/v1/traces').send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('idem-trace-svc') as Array>; + + // Should have exactly 1 dependency, not 2 + expect(deps).toHaveLength(1); + }); + + it('should handle payload with missing service.name gracefully', async () => { + const payload = { + resourceSpans: [ + { + resource: { attributes: [] }, + scopeSpans: [{ + spans: [{ + traceId: 'trace-1', + spanId: 'span-1', + name: 'test-span', + kind: 3, + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000050000000000', + attributes: [{ key: 'peer.service', value: { stringValue: 'target' } }], + }], + }], + }, + ], + }; + + const res = await request(app) + .post('/v1/traces') + .send(payload); + + // Should succeed with a warning, not crash + expect(res.status).toBe(200); + expect(res.body.partialSuccess.errorMessage).toContain('service.name'); + }); + + it('should store spans with correct duration_ms from nanosecond timestamps', async () => { + const startNano = '1700000000000000000'; + const endNano = '1700000000123000000'; // 123ms later (123,000,000 ns) + + const payload = buildTracePayload('duration-svc', [ + { + name: 'timed-call', + kind: 3, + startTimeUnixNano: startNano, + endTimeUnixNano: endNano, + attributes: [{ key: 'peer.service', value: { stringValue: 'tgt' } }], + }, + ]); + + await request(app).post('/v1/traces').send(payload); + + const spans = testDb + .prepare('SELECT * FROM spans WHERE service_name = ?') + .all('duration-svc') as Array>; + + expect(spans).toHaveLength(1); + expect(spans[0].duration_ms).toBe(123); + }); + + it('should store span attributes and resource attributes as JSON', async () => { + const payload = buildTracePayload('attrs-svc', [ + { + name: 'attr-span', + kind: 3, + attributes: [ + { key: 'peer.service', value: { stringValue: 'svc-target' } }, + { key: 'http.request.method', value: { stringValue: 'POST' } }, + ], + }, + ]); + + await request(app).post('/v1/traces').send(payload); + + const spans = testDb + .prepare('SELECT * FROM spans WHERE service_name = ?') + .all('attrs-svc') as Array>; + + expect(spans).toHaveLength(1); + // attributes should be stored as JSON string + expect(typeof spans[0].attributes).toBe('string'); + const attrs = JSON.parse(spans[0].attributes as string); + expect(attrs).toEqual(expect.arrayContaining([ + expect.objectContaining({ key: 'peer.service' }), + ])); + // resource_attributes should also be JSON + expect(typeof spans[0].resource_attributes).toBe('string'); + const resourceAttrs = JSON.parse(spans[0].resource_attributes as string); + expect(resourceAttrs).toEqual(expect.arrayContaining([ + expect.objectContaining({ key: 'service.name' }), + ])); + }); + + it('should handle multiple services in one payload', async () => { + const payload = { + resourceSpans: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'trace-svc-a' } }], + }, + scopeSpans: [{ + spans: [{ + traceId: 'trace-1', + spanId: 'span-a', + name: 'call-from-a', + kind: 3, + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000050000000000', + attributes: [{ key: 'peer.service', value: { stringValue: 'dep-from-a' } }], + }], + }], + }, + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'trace-svc-b' } }], + }, + scopeSpans: [{ + spans: [{ + traceId: 'trace-1', + spanId: 'span-b', + name: 'call-from-b', + kind: 3, + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000050000000000', + attributes: [{ key: 'peer.service', value: { stringValue: 'dep-from-b' } }], + }], + }], + }, + ], + }; + + const res = await request(app) + .post('/v1/traces') + .send(payload); + + expect(res.status).toBe(200); + + const services = testDb + .prepare('SELECT * FROM services WHERE team_id = ?') + .all(MOCK_TEAM_ID) as Array>; + + expect(services).toHaveLength(2); + const names = services.map((s) => s.name); + expect(names).toContain('trace-svc-a'); + expect(names).toContain('trace-svc-b'); + }); + + it('should auto-associate when trace from A calls registered service B', async () => { + // Pre-register service B in the same team + testDb.prepare( + `INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, poll_interval_ms, is_active, is_external) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ).run('svc-b', 'service-b', MOCK_TEAM_ID, '', 'otlp', 0, 1, 0); + + // Service A sends a trace with a CLIENT span calling service-b + const payload = buildTracePayload('service-a', [ + { + name: 'call-service-b', + kind: 3, + attributes: [ + { key: 'peer.service', value: { stringValue: 'service-b' } }, + { key: 'http.request.method', value: { stringValue: 'GET' } }, + ], + }, + ]); + + await request(app).post('/v1/traces').send(payload); + + const associations = testDb + .prepare('SELECT * FROM dependency_associations') + .all() as Array>; + + expect(associations).toHaveLength(1); + expect(associations[0].linked_service_id).toBe('svc-b'); + expect(associations[0].is_auto_suggested).toBe(1); + expect(associations[0].association_type).toBe('api_call'); + }); + + it('should not auto-associate when trace targets unregistered service', async () => { + // No pre-registered services — the target is unknown + const payload = buildTracePayload('service-a', [ + { + name: 'call-unknown', + kind: 3, + attributes: [ + { key: 'peer.service', value: { stringValue: 'unknown-service' } }, + ], + }, + ]); + + await request(app).post('/v1/traces').send(payload); + + const associations = testDb + .prepare('SELECT * FROM dependency_associations') + .all() as Array>; + + expect(associations).toHaveLength(0); + }); + + it('should not create duplicate associations on repeated trace pushes', async () => { + // Pre-register service B + testDb.prepare( + `INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, poll_interval_ms, is_active, is_external) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ).run('svc-b', 'service-b', MOCK_TEAM_ID, '', 'otlp', 0, 1, 0); + + const payload = buildTracePayload('service-a', [ + { + name: 'call-service-b', + kind: 3, + attributes: [{ key: 'peer.service', value: { stringValue: 'service-b' } }], + }, + ]); + + // Push twice + await request(app).post('/v1/traces').send(payload); + await request(app).post('/v1/traces').send(payload); + + const associations = testDb + .prepare('SELECT * FROM dependency_associations') + .all() as Array>; + + // Should have exactly 1, not 2 + expect(associations).toHaveLength(1); + }); + + it('should only create dependencies from CLIENT/PRODUCER spans, not SERVER/INTERNAL', async () => { + const payload = buildTracePayload('kind-filter-svc', [ + { + name: 'client-call', + spanId: 'span-client', + kind: 3, // CLIENT — should create dep + attributes: [{ key: 'peer.service', value: { stringValue: 'client-target' } }], + }, + { + name: 'producer-send', + spanId: 'span-producer', + kind: 4, // PRODUCER — should create dep + attributes: [{ key: 'messaging.system', value: { stringValue: 'kafka' } }], + }, + { + name: 'server-handle', + spanId: 'span-server', + kind: 2, // SERVER — should NOT create dep + attributes: [{ key: 'peer.service', value: { stringValue: 'server-target' } }], + }, + { + name: 'internal-work', + spanId: 'span-internal', + kind: 1, // INTERNAL — should NOT create dep + attributes: [{ key: 'peer.service', value: { stringValue: 'internal-target' } }], + }, + ]); + + await request(app).post('/v1/traces').send(payload); + + const deps = testDb + .prepare('SELECT * FROM dependencies WHERE service_id = (SELECT id FROM services WHERE name = ?)') + .all('kind-filter-svc') as Array>; + + expect(deps).toHaveLength(2); + const names = deps.map((d) => d.name); + expect(names).toContain('client-target'); + expect(names).toContain('kafka'); + expect(names).not.toContain('server-target'); + expect(names).not.toContain('internal-target'); + }); + }); +}); diff --git a/server/src/routes/otlp/traces.ts b/server/src/routes/otlp/traces.ts new file mode 100644 index 0000000..f495ba8 --- /dev/null +++ b/server/src/routes/otlp/traces.ts @@ -0,0 +1,185 @@ +import { Router, Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { TraceParser } from '../../services/polling/TraceParser'; +import { TraceDependencyBridge } from '../../services/polling/TraceDependencyBridge'; +import { AutoAssociator } from '../../services/polling/AutoAssociator'; +import { getDependencyUpsertService } from '../../services/polling/DependencyUpsertService'; +import { findOrCreateService } from '../../services/polling/otlpServiceResolver'; +import { HealthPollingService } from '../../services/polling'; +import { StatusChangeEvent, PollingEventType } from '../../services/polling/types'; +import { CreateSpanInput } from '../../db/types'; +import { OtlpResourceSpans } from '../../services/polling/otlp-types'; +import logger from '../../utils/logger'; + +const router = Router(); +const traceParser = new TraceParser(); +const bridge = new TraceDependencyBridge(); + +/** + * POST /v1/traces + * OTLP JSON trace receiver. Authenticated via API key (requireApiKeyAuth middleware). + * Stores ALL spans and extracts CLIENT/PRODUCER dependencies for discovery. + */ +router.post('/', (req: Request, res: Response): void => { + const teamId = req.apiKeyTeamId; + + if (!teamId) { + res.status(401).json({ error: 'Missing team context' }); + return; + } + + // Validate basic OTLP structure + const data = req.body; + if (!data || typeof data !== 'object' || !Array.isArray(data.resourceSpans)) { + logger.warn('OTLP trace parse error: invalid payload structure'); + res.status(400).json({ + partialSuccess: { + rejectedDataPoints: -1, + errorMessage: 'Invalid OTLP payload: expected object with resourceSpans array', + }, + }); + return; + } + + const resourceSpans: OtlpResourceSpans[] = data.resourceSpans; + const stores = getStores(); + const upsertService = getDependencyUpsertService(); + const warnings: string[] = []; + let totalRejected = 0; + const allChanges: StatusChangeEvent[] = []; + + for (const rs of resourceSpans) { + try { + const serviceName = traceParser.extractServiceName(rs); + if (!serviceName) { + warnings.push('Skipping resourceSpans entry: missing service.name resource attribute'); + continue; + } + + // Find or auto-register the service + const service = findOrCreateService(stores, teamId, serviceName, warnings); + + // Store ALL spans (full span storage for future trace timeline views) + const spanInputs = buildSpanInputs(rs, serviceName, teamId); + if (spanInputs.length > 0) { + stores.spans.bulkInsert(spanInputs); + } + + // Parse CLIENT/PRODUCER spans for dependency discovery + const result = traceParser.parseResourceSpans(rs); + warnings.push(...traceParser.lastWarnings); + + if (result.dependencies.length > 0) { + // Convert trace dependencies to ProactiveDepsStatus for the upsert pipeline + const depsStatus = bridge.bridgeToDepsStatus(result.dependencies); + + // Upsert dependencies + const changes = upsertService.upsert(service, depsStatus); + allChanges.push(...changes); + + // Auto-associate trace-discovered dependencies with registered services + const autoAssociator = new AutoAssociator(stores); + autoAssociator.processDiscoveredDependencies(service, depsStatus, teamId); + } + + // Update poll result on the service + stores.services.updatePollResult(service.id, true, undefined, warnings.length > 0 ? warnings : undefined); + } catch (err) { + const message = err instanceof Error ? err.message : 'Unknown error'; + logger.error({ err }, 'OTLP trace processing failed for resourceSpans entry'); + warnings.push(message); + totalRejected++; + } + } + + // Emit status change events to the polling service for alert processing + if (allChanges.length > 0) { + try { + const pollingService = HealthPollingService.getInstance(); + for (const change of allChanges) { + pollingService.emit(PollingEventType.STATUS_CHANGE, change); + } + } catch { + // Polling service may not be initialized in tests — non-critical + } + } + + res.status(200).json({ + partialSuccess: { + rejectedDataPoints: totalRejected, + errorMessage: warnings.length > 0 ? warnings.join('; ') : '', + }, + }); +}); + +/** + * Convert all spans from a resourceSpans entry into CreateSpanInput[]. + * Flattens scopeSpans → spans, precomputing duration_ms and serializing attributes. + */ +function buildSpanInputs( + rs: OtlpResourceSpans, + serviceName: string, + teamId: string, +): CreateSpanInput[] { + const inputs: CreateSpanInput[] = []; + const resourceAttrsJson = rs.resource?.attributes + ? JSON.stringify(rs.resource.attributes) + : null; + + if (!Array.isArray(rs.scopeSpans)) return inputs; + + for (const ss of rs.scopeSpans) { + if (!Array.isArray(ss.spans)) continue; + + for (const span of ss.spans) { + const durationMs = computeDurationMs(span.startTimeUnixNano, span.endTimeUnixNano); + const startTime = nanoToIso(span.startTimeUnixNano); + const endTime = nanoToIso(span.endTimeUnixNano); + + inputs.push({ + trace_id: span.traceId, + span_id: span.spanId, + parent_span_id: span.parentSpanId ?? null, + service_name: serviceName, + team_id: teamId, + name: span.name, + kind: span.kind ?? 0, + start_time: startTime, + end_time: endTime, + duration_ms: durationMs, + status_code: span.status?.code ?? 0, + status_message: span.status?.message ?? null, + attributes: span.attributes ? JSON.stringify(span.attributes) : null, + resource_attributes: resourceAttrsJson, + }); + } + } + + return inputs; +} + +/** + * Compute duration in milliseconds from nanosecond timestamps. + */ +function computeDurationMs(startNano: string, endNano: string): number { + try { + const durationNanos = BigInt(endNano) - BigInt(startNano); + return Number(durationNanos / BigInt(1_000_000)); + } catch { + return 0; + } +} + +/** + * Convert nanosecond Unix timestamp to ISO 8601 string. + */ +function nanoToIso(nanoTimestamp: string): string { + try { + const ms = Number(BigInt(nanoTimestamp) / BigInt(1_000_000)); + return new Date(ms).toISOString(); + } catch { + return new Date().toISOString(); + } +} + +export default router; diff --git a/server/src/routes/services/create.ts b/server/src/routes/services/create.ts index f2188d9..51d2301 100644 --- a/server/src/routes/services/create.ts +++ b/server/src/routes/services/create.ts @@ -26,6 +26,7 @@ export function createService(req: Request, res: Response): void { metrics_endpoint: validated.metrics_endpoint, schema_config: validated.schema_config, poll_interval_ms: validated.poll_interval_ms, + health_endpoint_format: validated.health_endpoint_format, }); // Start polling for the new service (is_active defaults to 1) diff --git a/server/src/routes/services/index.ts b/server/src/routes/services/index.ts index 56c8a67..7f61257 100644 --- a/server/src/routes/services/index.ts +++ b/server/src/routes/services/index.ts @@ -9,6 +9,7 @@ import { deleteService } from './delete'; import { pollServiceNow } from './poll'; import { testSchema } from './testSchema'; import { getServicePollHistory } from './pollHistory'; +import { listDiscovered } from '../dependencies/listDiscovered'; const router = Router(); @@ -29,4 +30,7 @@ router.get('/:id/poll-history', requireAuth, getServicePollHistory); // Trigger immediate poll requires team membership (not just lead) router.post('/:id/poll', requireServiceTeamAccess, pollServiceNow); +// GET /api/services/:serviceId/discovered-dependencies - List trace-discovered dependencies +router.get('/:serviceId/discovered-dependencies', requireAuth, listDiscovered); + export default router; diff --git a/server/src/routes/services/services.test.ts b/server/src/routes/services/services.test.ts index 00bf567..b5f8f9b 100644 --- a/server/src/routes/services/services.test.ts +++ b/server/src/routes/services/services.test.ts @@ -78,6 +78,7 @@ describe('Services API', () => { is_active INTEGER NOT NULL DEFAULT 1, is_external INTEGER NOT NULL DEFAULT 0, description TEXT, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', last_poll_success INTEGER, last_poll_error TEXT, poll_warnings TEXT, diff --git a/server/src/routes/services/testSchema.test.ts b/server/src/routes/services/testSchema.test.ts index 189da29..a75c3d1 100644 --- a/server/src/routes/services/testSchema.test.ts +++ b/server/src/routes/services/testSchema.test.ts @@ -554,4 +554,99 @@ describe('POST /api/services/test-schema', () => { expect(response.body.dependencies[1].name).toBe('db'); expect(response.body.dependencies[1].healthy).toBe(false); }); + + // --- Format-aware tests --- + + it('should return error for OTLP format', async () => { + const response = await request(app) + .post('/api/services/test-schema') + .send({ url: validUrl, format: 'otlp' }); + + expect(response.status).toBe(400); + expect(response.body.error).toMatch(/OTLP services receive pushed metrics/); + }); + + it('should fetch with Accept: text/plain for prometheus format', async () => { + const promText = [ + 'dependency_health_status{name="postgres"} 0', + 'dependency_health_healthy{name="postgres"} 1', + 'dependency_health_latency_ms{name="postgres"} 12', + ].join('\n'); + + mockFetch.mockResolvedValue({ + ok: true, + text: async () => promText, + }); + + const response = await request(app) + .post('/api/services/test-schema') + .send({ url: validUrl, format: 'prometheus' }); + + expect(response.status).toBe(200); + expect(response.body.success).toBe(true); + expect(response.body.dependencies).toHaveLength(1); + expect(response.body.dependencies[0].name).toBe('postgres'); + expect(response.body.dependencies[0].healthy).toBe(true); + expect(response.body.dependencies[0].latency_ms).toBe(12); + + expect(mockFetch).toHaveBeenCalledWith( + validUrl, + expect.objectContaining({ + headers: expect.objectContaining({ + Accept: 'text/plain; version=0.0.4', + }), + }) + ); + }); + + it('should not require schema_config for prometheus format', async () => { + mockFetch.mockResolvedValue({ + ok: true, + text: async () => 'dependency_health_status{name="db"} 0\ndependency_health_healthy{name="db"} 1\n', + }); + + const response = await request(app) + .post('/api/services/test-schema') + .send({ url: validUrl, format: 'prometheus' }); + + expect(response.status).toBe(200); + expect(response.body.success).toBe(true); + }); + + it('should still require schema_config for schema format', async () => { + const response = await request(app) + .post('/api/services/test-schema') + .send({ url: validUrl, format: 'schema' }); + + expect(response.status).toBe(400); + expect(response.body.error).toMatch(/schema_config is required/i); + }); + + it('should default format to schema for backward compatibility', async () => { + // Without format field, should still work as before (requiring schema_config) + const response = await request(app) + .post('/api/services/test-schema') + .send({ url: validUrl }); + + expect(response.status).toBe(400); + expect(response.body.error).toMatch(/schema_config is required/i); + }); + + it('should not include schema-specific warnings for prometheus format', async () => { + mockFetch.mockResolvedValue({ + ok: true, + text: async () => 'dependency_health_status{name="db"} 0\ndependency_health_healthy{name="db"} 1\n', + }); + + const response = await request(app) + .post('/api/services/test-schema') + .send({ url: validUrl, format: 'prometheus' }); + + expect(response.status).toBe(200); + // Should NOT have schema-specific warnings like "No latency field mapping configured" + const schemaWarnings = (response.body.warnings || []).filter( + (w: string) => w.includes('field mapping configured') + ); + expect(schemaWarnings).toHaveLength(0); + }); }); diff --git a/server/src/routes/services/testSchema.ts b/server/src/routes/services/testSchema.ts index 56f356f..c611f89 100644 --- a/server/src/routes/services/testSchema.ts +++ b/server/src/routes/services/testSchema.ts @@ -1,9 +1,10 @@ import { Request, Response } from 'express'; import { getStores } from '../../stores'; -import { validateSchemaConfig } from '../../utils/validation'; +import { validateSchemaConfig, validateMetricSchemaConfig } from '../../utils/validation'; import { validateUrlNotPrivate, validateUrlHostname } from '../../utils/ssrf'; import { DependencyParser } from '../../services/polling/DependencyParser'; -import { SchemaMapping } from '../../db/types'; +import { PrometheusParser } from '../../services/polling/PrometheusParser'; +import { SchemaMapping, HealthEndpointFormat, MetricSchemaConfig } from '../../db/types'; import { ValidationError, ForbiddenError, sendErrorResponse } from '../../utils/errors'; const TEST_SCHEMA_TIMEOUT_MS = 10_000; @@ -14,6 +15,12 @@ const TEST_SCHEMA_TIMEOUT_MS = 10_000; * Tests a schema mapping against a live health endpoint URL. * Returns parsed dependency results and any warnings. * Does NOT store anything — purely a preview/test operation. + * + * Supports format-aware testing: + * - 'schema' (default): Uses SchemaMapper with provided schema_config + * - 'prometheus': Fetches with text/plain Accept, parses Prometheus exposition format + * - 'otlp': Returns error (push-only, cannot be tested via URL) + * - 'default': Uses schema_config if provided for backward compat */ export async function testSchema(req: Request, res: Response): Promise { try { @@ -29,7 +36,16 @@ export async function testSchema(req: Request, res: Response): Promise { } } - const { url, schema_config } = req.body; + const { url, schema_config, format } = req.body; + const effectiveFormat: HealthEndpointFormat = format ?? 'schema'; + + // OTLP services are push-only — cannot be tested via URL + if (effectiveFormat === 'otlp') { + throw new ValidationError( + 'OTLP services receive pushed metrics and cannot be tested via URL', + 'format' + ); + } // Validate URL is provided if (!url || typeof url !== 'string') { @@ -43,19 +59,24 @@ export async function testSchema(req: Request, res: Response): Promise { throw new ValidationError('url must be a valid URL', 'url'); } - // Validate schema_config is provided - if (schema_config === undefined || schema_config === null) { - throw new ValidationError('schema_config is required', 'schema_config'); - } + // schema_config is required only for 'schema' format (or default without explicit format) + let schemaConfig: SchemaMapping | null = null; + if (effectiveFormat === 'schema') { + if (schema_config === undefined || schema_config === null) { + throw new ValidationError('schema_config is required', 'schema_config'); + } - // Validate schema_config structure (returns JSON string) - const validatedSchemaJson = validateSchemaConfig(schema_config); - const schemaConfig: SchemaMapping = JSON.parse(validatedSchemaJson); + // Validate schema_config structure (returns JSON string) + const validatedSchemaJson = validateSchemaConfig(schema_config); + schemaConfig = JSON.parse(validatedSchemaJson); + } // SSRF validation — sync hostname check + async DNS resolution validateUrlHostname(url); await validateUrlNotPrivate(url); + const isPrometheus = effectiveFormat === 'prometheus'; + // Fetch the health endpoint const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), TEST_SCHEMA_TIMEOUT_MS); @@ -64,7 +85,7 @@ export async function testSchema(req: Request, res: Response): Promise { try { const response = await fetch(url, { signal: controller.signal, - headers: { Accept: 'application/json' }, + headers: { Accept: isPrometheus ? 'text/plain; version=0.0.4' : 'application/json' }, }); if (!response.ok) { @@ -74,7 +95,7 @@ export async function testSchema(req: Request, res: Response): Promise { ); } - responseData = await response.json(); + responseData = isPrometheus ? await response.text() : await response.json(); } catch (error) { if (error instanceof ValidationError) throw error; @@ -88,13 +109,27 @@ export async function testSchema(req: Request, res: Response): Promise { clearTimeout(timeout); } - // Parse using the schema mapping + // Parse MetricSchemaConfig for prometheus format if provided + let metricConfig: MetricSchemaConfig | undefined; + if (isPrometheus && schema_config !== undefined && schema_config !== null) { + const validatedJson = validateMetricSchemaConfig(schema_config); + metricConfig = JSON.parse(validatedJson); + } + + // Parse based on format const parser = new DependencyParser(); const warnings: string[] = []; let dependencies; try { - dependencies = parser.parse(responseData, schemaConfig); + if (isPrometheus) { + const promParser = new PrometheusParser(); + dependencies = promParser.parse(responseData as string, metricConfig); + warnings.push(...promParser.lastWarnings); + } else { + dependencies = parser.parse(responseData, schemaConfig); + warnings.push(...parser.lastWarnings); + } } catch (error) { const message = error instanceof Error ? error.message : 'Unknown error'; res.json({ @@ -105,30 +140,29 @@ export async function testSchema(req: Request, res: Response): Promise { return; } - // Include any schema mapping warnings (skipped items, etc.) - warnings.push(...parser.lastWarnings); - - // Collect warnings about missing optional fields - if (!schemaConfig.fields.latency) { - warnings.push('No latency field mapping configured — latency data will not be captured'); - } - if (!schemaConfig.fields.impact) { - warnings.push('No impact field mapping configured — impact data will not be captured'); - } - if (!schemaConfig.fields.description) { - warnings.push('No description field mapping configured — description data will not be captured'); - } - if (!schemaConfig.fields.checkDetails) { - warnings.push('No checkDetails field mapping configured — check details data will not be captured'); - } - if (!schemaConfig.fields.contact) { - warnings.push('No contact field mapping configured — contact data will not be captured'); - } + // Collect warnings about missing optional fields (schema format only) + if (effectiveFormat === 'schema' && schemaConfig) { + if (!schemaConfig.fields.latency) { + warnings.push('No latency field mapping configured — latency data will not be captured'); + } + if (!schemaConfig.fields.impact) { + warnings.push('No impact field mapping configured — impact data will not be captured'); + } + if (!schemaConfig.fields.description) { + warnings.push('No description field mapping configured — description data will not be captured'); + } + if (!schemaConfig.fields.checkDetails) { + warnings.push('No checkDetails field mapping configured — check details data will not be captured'); + } + if (!schemaConfig.fields.contact) { + warnings.push('No contact field mapping configured — contact data will not be captured'); + } - // Check for entries with missing optional data - for (const dep of dependencies) { - if (schemaConfig.fields.latency && dep.health.latency === 0) { - warnings.push(`Dependency "${dep.name}": latency field resolved to 0 or was not found`); + // Check for entries with missing optional data + for (const dep of dependencies) { + if (schemaConfig.fields.latency && dep.health.latency === 0) { + warnings.push(`Dependency "${dep.name}": latency field resolved to 0 or was not found`); + } } } diff --git a/server/src/routes/services/update.ts b/server/src/routes/services/update.ts index ba369ff..1ad763b 100644 --- a/server/src/routes/services/update.ts +++ b/server/src/routes/services/update.ts @@ -40,13 +40,15 @@ export function updateService(req: Request, res: Response): void { schema_config: validated.schema_config, poll_interval_ms: validated.poll_interval_ms, is_active: validated.is_active, + health_endpoint_format: validated.health_endpoint_format, }); - // Update polling service if is_active, health_endpoint, or poll_interval_ms changed + // Update polling service if is_active, health_endpoint, poll_interval_ms, or format changed if ( validated.is_active !== undefined || validated.health_endpoint !== undefined || - validated.poll_interval_ms !== undefined + validated.poll_interval_ms !== undefined || + validated.health_endpoint_format !== undefined ) { const pollingService = HealthPollingService.getInstance(); const newIsActive = diff --git a/server/src/routes/teams/apiKeyRateLimit.test.ts b/server/src/routes/teams/apiKeyRateLimit.test.ts new file mode 100644 index 0000000..54ead7c --- /dev/null +++ b/server/src/routes/teams/apiKeyRateLimit.test.ts @@ -0,0 +1,413 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +interface TestUser { + id: string; + email: string; + name: string; + role: string; + is_active: number; + created_at: string; + updated_at: string; + oidc_subject: string | null; + password_hash: string | null; +} + +const adminUser: TestUser = { + id: 'admin-1', + email: 'admin@test.com', + name: 'Admin User', + role: 'admin', + is_active: 1, + created_at: '2024-01-01T00:00:00.000Z', + updated_at: '2024-01-01T00:00:00.000Z', + oidc_subject: null, + password_hash: null, +}; + +const leadUser: TestUser = { + id: 'lead-1', + email: 'lead@test.com', + name: 'Lead User', + role: 'user', + is_active: 1, + created_at: '2024-01-01T00:00:00.000Z', + updated_at: '2024-01-01T00:00:00.000Z', + oidc_subject: null, + password_hash: null, +}; + +const memberUser: TestUser = { + id: 'member-1', + email: 'member@test.com', + name: 'Member User', + role: 'user', + is_active: 1, + created_at: '2024-01-01T00:00:00.000Z', + updated_at: '2024-01-01T00:00:00.000Z', + oidc_subject: null, + password_hash: null, +}; + +let currentUser: TestUser = adminUser; + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireAdmin: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireTeamLead: jest.fn( + ( + req: Record, + res: { status: (code: number) => { json: (body: unknown) => void } }, + next: () => void, + ) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + + if (currentUser.role === 'admin') { + next(); + return; + } + + const teamId = (req.params as Record).id; + const membership = testDb + .prepare('SELECT * FROM team_members WHERE team_id = ? AND user_id = ?') + .get(teamId, currentUser.id) as { role: string } | undefined; + if (!membership || membership.role !== 'lead') { + res.status(403).json({ error: 'Access denied' }); + return; + } + next(); + }, + ), +})); + +import teamRouter from './index'; + +const app = express(); +app.use(express.json()); +app.use('/api/teams', teamRouter); + +describe('Team API Key Rate Limit Routes', () => { + const teamId = 'team-1'; + const otherTeamId = 'team-2'; + + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE team_members ( + team_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (team_id, user_id), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT NOT NULL DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms REAL, + contact TEXT, + contact_override TEXT, + impact_override TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + last_checked TEXT, + last_status_change TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE service_poll_history ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + error TEXT, + recorded_at TEXT NOT NULL, + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + `); + + const insertUser = testDb.prepare( + 'INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)', + ); + insertUser.run(adminUser.id, adminUser.email, adminUser.name, adminUser.role); + insertUser.run(leadUser.id, leadUser.email, leadUser.name, leadUser.role); + insertUser.run(memberUser.id, memberUser.email, memberUser.name, memberUser.role); + + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(teamId, 'Test Team', 'TEST'); + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(otherTeamId, 'Other Team', 'OTHER'); + + testDb.prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)').run(teamId, leadUser.id, 'lead'); + testDb.prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)').run(teamId, memberUser.id, 'member'); + }); + + afterAll(() => { + testDb.close(); + }); + + beforeEach(() => { + currentUser = adminUser; + testDb.exec('DELETE FROM team_api_keys'); + }); + + function insertApiKey( + id: string, + teamId: string, + opts: { rate_limit_rpm?: number | null; rate_limit_admin_locked?: number } = {}, + ) { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, rate_limit_admin_locked, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)', + ).run( + id, teamId, `Key ${id}`, `hash-${id}`, 'dps_test', + opts.rate_limit_rpm ?? null, + opts.rate_limit_admin_locked ?? 0, + '2026-01-01T00:00:00Z', + ); + } + + describe('PATCH /api/teams/:id/api-keys/:keyId/rate-limit', () => { + it('should allow team lead to update rate limit', async () => { + currentUser = leadUser; + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_rpm).toBe(5000); + expect(res.body.key_hash).toBeUndefined(); + + const row = testDb.prepare('SELECT rate_limit_rpm FROM team_api_keys WHERE id = ?').get('key-1') as { rate_limit_rpm: number }; + expect(row.rate_limit_rpm).toBe(5000); + }); + + it('should allow admin to update rate limit', async () => { + currentUser = adminUser; + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 10000 }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_rpm).toBe(10000); + }); + + it('should deny regular member from updating rate limit', async () => { + currentUser = memberUser; + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(403); + }); + + it('should return 404 for key belonging to a different team', async () => { + currentUser = adminUser; + insertApiKey('key-other', otherTeamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-other/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(404); + expect(res.body.error).toBe('API key not found'); + }); + + it('should return 404 for nonexistent key', async () => { + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/nonexistent/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(404); + }); + + it('should return 403 when key is admin-locked', async () => { + insertApiKey('key-locked', teamId, { rate_limit_admin_locked: 1 }); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-locked/rate-limit`) + .send({ rate_limit_rpm: 5000 }); + + expect(res.status).toBe(403); + expect(res.body.error).toBe('Rate limit locked by admin'); + }); + + it('should return 400 when setting rate_limit_rpm to 0 (unlimited)', async () => { + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 0 }); + + expect(res.status).toBe(400); + expect(res.body.error).toContain('Unlimited (0) can only be set by admins'); + }); + + it('should return 400 when rate_limit_rpm exceeds maximum', async () => { + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 2_000_000 }); + + expect(res.status).toBe(400); + expect(res.body.error).toContain('exceeds maximum of 1,500,000'); + }); + + it('should return 400 for negative rate_limit_rpm', async () => { + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: -100 }); + + expect(res.status).toBe(400); + expect(res.body.error).toContain('positive integer or null'); + }); + + it('should return 400 for non-integer rate_limit_rpm', async () => { + insertApiKey('key-1', teamId); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: 5000.5 }); + + expect(res.status).toBe(400); + }); + + it('should allow resetting to default with null', async () => { + insertApiKey('key-1', teamId, { rate_limit_rpm: 5000 }); + + const res = await request(app) + .patch(`/api/teams/${teamId}/api-keys/key-1/rate-limit`) + .send({ rate_limit_rpm: null }); + + expect(res.status).toBe(200); + expect(res.body.rate_limit_rpm).toBeNull(); + + const row = testDb.prepare('SELECT rate_limit_rpm FROM team_api_keys WHERE id = ?').get('key-1') as { rate_limit_rpm: number | null }; + expect(row.rate_limit_rpm).toBeNull(); + }); + }); +}); diff --git a/server/src/routes/teams/apiKeyRateLimit.ts b/server/src/routes/teams/apiKeyRateLimit.ts new file mode 100644 index 0000000..87063bd --- /dev/null +++ b/server/src/routes/teams/apiKeyRateLimit.ts @@ -0,0 +1,57 @@ +import { Router, Request, Response } from 'express'; +import { requireTeamLead } from '../../auth'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; +import { evictBucket } from '../../middleware/perKeyRateLimit'; + +const router = Router({ mergeParams: true }); + +/** + * PATCH /api/teams/:id/api-keys/:keyId/rate-limit + * Team lead endpoint to update their own key's rate limit. + */ +router.patch('/:keyId/rate-limit', requireTeamLead, (req: Request, res: Response): void => { + try { + const teamId = req.params.id; + const keyId = req.params.keyId; + const stores = getStores(); + + const key = stores.teamApiKeys.findById(keyId); + if (!key || key.team_id !== teamId) { + res.status(404).json({ error: 'API key not found' }); + return; + } + + if (key.rate_limit_admin_locked === 1) { + res.status(403).json({ error: 'Rate limit locked by admin' }); + return; + } + + const { rate_limit_rpm } = req.body; + + if (rate_limit_rpm !== null) { + if (rate_limit_rpm === 0) { + res.status(400).json({ error: 'Unlimited (0) can only be set by admins' }); + return; + } + if (typeof rate_limit_rpm !== 'number' || !Number.isInteger(rate_limit_rpm) || rate_limit_rpm < 1) { + res.status(400).json({ error: 'rate_limit_rpm must be a positive integer or null' }); + return; + } + if (rate_limit_rpm > 1_500_000) { + res.status(400).json({ error: 'rate_limit_rpm exceeds maximum of 1,500,000' }); + return; + } + } + + const updated = stores.teamApiKeys.updateRateLimit(keyId, rate_limit_rpm); + evictBucket(keyId); + + const { key_hash: _hash, ...sanitized } = updated; + res.json(sanitized); + } catch (error) { + sendErrorResponse(res, error, 'updating API key rate limit'); + } +}); + +export default router; diff --git a/server/src/routes/teams/apiKeyUsage.test.ts b/server/src/routes/teams/apiKeyUsage.test.ts new file mode 100644 index 0000000..726e181 --- /dev/null +++ b/server/src/routes/teams/apiKeyUsage.test.ts @@ -0,0 +1,391 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +interface TestUser { + id: string; + email: string; + name: string; + role: string; +} + +const adminUser: TestUser = { + id: 'admin-1', + email: 'admin@test.com', + name: 'Admin User', + role: 'admin', +}; + +const leadUser: TestUser = { + id: 'lead-1', + email: 'lead@test.com', + name: 'Lead User', + role: 'user', +}; + +const memberUser: TestUser = { + id: 'member-1', + email: 'member@test.com', + name: 'Member User', + role: 'user', +}; + +let currentUser: TestUser = adminUser; + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireAdmin: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireTeamLead: jest.fn( + ( + req: Record, + res: { status: (code: number) => { json: (body: unknown) => void } }, + next: () => void, + ) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + + if (currentUser.role === 'admin') { + next(); + return; + } + + const teamId = (req.params as Record).id; + const membership = testDb + .prepare('SELECT * FROM team_members WHERE team_id = ? AND user_id = ?') + .get(teamId, currentUser.id) as { role: string } | undefined; + if (!membership || membership.role !== 'lead') { + res.status(403).json({ error: 'Access denied' }); + return; + } + next(); + }, + ), +})); + +import teamRouter from './index'; + +const app = express(); +app.use(express.json()); +app.use('/api/teams', teamRouter); + +describe('Team API Key Usage Routes', () => { + const teamId = 'team-1'; + const otherTeamId = 'team-2'; + + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE team_members ( + team_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (team_id, user_id), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT NOT NULL DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms REAL, + contact TEXT, + contact_override TEXT, + impact_override TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + last_checked TEXT, + last_status_change TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE service_poll_history ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + error TEXT, + recorded_at TEXT NOT NULL, + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + `); + + const insertUser = testDb.prepare( + 'INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)', + ); + insertUser.run(adminUser.id, adminUser.email, adminUser.name, adminUser.role); + insertUser.run(leadUser.id, leadUser.email, leadUser.name, leadUser.role); + insertUser.run(memberUser.id, memberUser.email, memberUser.name, memberUser.role); + + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(teamId, 'Test Team', 'TEST'); + testDb.prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)').run(otherTeamId, 'Other Team', 'OTHER'); + + testDb.prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)').run(teamId, leadUser.id, 'lead'); + testDb.prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)').run(teamId, memberUser.id, 'member'); + }); + + afterAll(() => { + testDb.close(); + }); + + beforeEach(() => { + currentUser = adminUser; + testDb.exec('DELETE FROM team_api_keys'); + testDb.exec('DELETE FROM api_key_usage_buckets'); + }); + + function insertApiKey(id: string, teamId: string) { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_at) VALUES (?, ?, ?, ?, ?, ?)', + ).run(id, teamId, `Key ${id}`, `hash-${id}`, 'dps_test', '2026-01-01T00:00:00Z'); + } + + function insertUsageBucket( + apiKeyId: string, + bucketStart: string, + granularity: 'minute' | 'hour', + pushCount: number, + rejectedCount: number = 0, + ) { + testDb.prepare( + 'INSERT INTO api_key_usage_buckets (api_key_id, bucket_start, granularity, push_count, rejected_count) VALUES (?, ?, ?, ?, ?)', + ).run(apiKeyId, bucketStart, granularity, pushCount, rejectedCount); + } + + describe('GET /api/teams/:id/api-keys/:keyId/usage', () => { + it('should return usage buckets for a valid key and time range', async () => { + insertApiKey('key-1', teamId); + const now = new Date(); + const bucketStart = new Date(now.getTime() - 60 * 60 * 1000).toISOString().slice(0, 16) + ':00'; + insertUsageBucket('key-1', bucketStart, 'minute', 100, 5); + + const from = new Date(now.getTime() - 2 * 60 * 60 * 1000).toISOString(); + const to = now.toISOString(); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'minute', from, to }); + + expect(res.status).toBe(200); + expect(res.body.api_key_id).toBe('key-1'); + expect(res.body.granularity).toBe('minute'); + expect(res.body.buckets).toHaveLength(1); + expect(res.body.buckets[0].push_count).toBe(100); + expect(res.body.buckets[0].rejected_count).toBe(5); + }); + + it('should return empty buckets when no usage data exists', async () => { + insertApiKey('key-1', teamId); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'minute' }); + + expect(res.status).toBe(200); + expect(res.body.buckets).toEqual([]); + }); + + it('should return only minute-granularity rows when granularity=minute', async () => { + insertApiKey('key-1', teamId); + const now = new Date(); + const minuteBucket = new Date(now.getTime() - 30 * 60 * 1000).toISOString().slice(0, 16) + ':00'; + const hourBucket = new Date(now.getTime() - 30 * 60 * 1000).toISOString().slice(0, 13) + ':00:00'; + + insertUsageBucket('key-1', minuteBucket, 'minute', 50); + insertUsageBucket('key-1', hourBucket, 'hour', 200); + + const from = new Date(now.getTime() - 2 * 60 * 60 * 1000).toISOString(); + const to = now.toISOString(); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'minute', from, to }); + + expect(res.status).toBe(200); + expect(res.body.buckets).toHaveLength(1); + expect(res.body.buckets[0].push_count).toBe(50); + }); + + it('should return only hour-granularity rows when granularity=hour', async () => { + insertApiKey('key-1', teamId); + const now = new Date(); + const minuteBucket = new Date(now.getTime() - 30 * 60 * 1000).toISOString().slice(0, 16) + ':00'; + const hourBucket = new Date(now.getTime() - 30 * 60 * 1000).toISOString().slice(0, 13) + ':00:00'; + + insertUsageBucket('key-1', minuteBucket, 'minute', 50); + insertUsageBucket('key-1', hourBucket, 'hour', 200); + + const from = new Date(now.getTime() - 2 * 60 * 60 * 1000).toISOString(); + const to = now.toISOString(); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'hour', from, to }); + + expect(res.status).toBe(200); + expect(res.body.buckets).toHaveLength(1); + expect(res.body.buckets[0].push_count).toBe(200); + }); + + it('should return 403 when key belongs to a different team', async () => { + insertApiKey('key-other', otherTeamId); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-other/usage`) + .query({ granularity: 'minute' }); + + expect(res.status).toBe(403); + expect(res.body.error).toContain('does not belong to this team'); + }); + + it('should deny regular member from accessing usage', async () => { + currentUser = memberUser; + insertApiKey('key-1', teamId); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'minute' }); + + expect(res.status).toBe(403); + }); + + it('should allow team lead to access usage', async () => { + currentUser = leadUser; + insertApiKey('key-1', teamId); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'minute' }); + + expect(res.status).toBe(200); + expect(res.body.api_key_id).toBe('key-1'); + }); + + it('should use default time range when from/to are omitted', async () => { + insertApiKey('key-1', teamId); + + const res = await request(app) + .get(`/api/teams/${teamId}/api-keys/key-1/usage`) + .query({ granularity: 'minute' }); + + expect(res.status).toBe(200); + expect(res.body.from).toBeDefined(); + expect(res.body.to).toBeDefined(); + expect(res.body.granularity).toBe('minute'); + }); + }); +}); diff --git a/server/src/routes/teams/apiKeyUsage.ts b/server/src/routes/teams/apiKeyUsage.ts new file mode 100644 index 0000000..384a994 --- /dev/null +++ b/server/src/routes/teams/apiKeyUsage.ts @@ -0,0 +1,46 @@ +import { Router, Request, Response } from 'express'; +import { requireTeamLead } from '../../auth'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; + +const router = Router({ mergeParams: true }); + +/** + * GET /api/teams/:id/api-keys/:keyId/usage + * Team endpoint to fetch a key's usage time series. + */ +router.get('/:keyId/usage', requireTeamLead, (req: Request, res: Response): void => { + try { + const teamId = req.params.id; + const keyId = req.params.keyId; + const stores = getStores(); + + const key = stores.teamApiKeys.findById(keyId); + if (!key || key.team_id !== teamId) { + res.status(403).json({ error: 'API key does not belong to this team' }); + return; + } + + const granularity = (req.query.granularity as 'minute' | 'hour') || 'minute'; + if (granularity !== 'minute' && granularity !== 'hour') { + res.status(400).json({ error: 'granularity must be "minute" or "hour"' }); + return; + } + + const now = new Date(); + const defaultFrom = granularity === 'minute' + ? new Date(now.getTime() - 24 * 60 * 60 * 1000) + : new Date(now.getTime() - 30 * 24 * 60 * 60 * 1000); + + const from = (req.query.from as string) || defaultFrom.toISOString(); + const to = (req.query.to as string) || now.toISOString(); + + const buckets = stores.apiKeyUsage.getBuckets(keyId, granularity, from, to); + + res.json({ api_key_id: keyId, granularity, from, to, buckets }); + } catch (error) { + sendErrorResponse(res, error, 'fetching API key usage'); + } +}); + +export default router; diff --git a/server/src/routes/teams/apiKeys.test.ts b/server/src/routes/teams/apiKeys.test.ts new file mode 100644 index 0000000..3ac4cb8 --- /dev/null +++ b/server/src/routes/teams/apiKeys.test.ts @@ -0,0 +1,362 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +interface TestUser { + id: string; + email: string; + name: string; + role: string; + is_active: number; + created_at: string; + updated_at: string; + oidc_subject: string | null; + password_hash: string | null; +} + +const adminUser: TestUser = { + id: 'admin-1', + email: 'admin@test.com', + name: 'Admin User', + role: 'admin', + is_active: 1, + created_at: '2024-01-01T00:00:00.000Z', + updated_at: '2024-01-01T00:00:00.000Z', + oidc_subject: null, + password_hash: null, +}; + +const leadUser: TestUser = { + id: 'lead-1', + email: 'lead@test.com', + name: 'Lead User', + role: 'user', + is_active: 1, + created_at: '2024-01-01T00:00:00.000Z', + updated_at: '2024-01-01T00:00:00.000Z', + oidc_subject: null, + password_hash: null, +}; + +const memberUser: TestUser = { + id: 'member-1', + email: 'member@test.com', + name: 'Member User', + role: 'user', + is_active: 1, + created_at: '2024-01-01T00:00:00.000Z', + updated_at: '2024-01-01T00:00:00.000Z', + oidc_subject: null, + password_hash: null, +}; + +let currentUser: TestUser = adminUser; + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireAdmin: jest.fn((req: Record, _res: unknown, next: () => void) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + next(); + }), + requireTeamLead: jest.fn( + ( + req: Record, + res: { status: (code: number) => { json: (body: unknown) => void } }, + next: () => void, + ) => { + req.user = currentUser; + req.session = { userId: currentUser.id }; + + if (currentUser.role === 'admin') { + next(); + return; + } + + const teamId = (req.params as Record).id; + const membership = testDb + .prepare('SELECT * FROM team_members WHERE team_id = ? AND user_id = ?') + .get(teamId, currentUser.id) as { role: string } | undefined; + if (!membership || membership.role !== 'lead') { + res.status(403).json({ error: 'Access denied' }); + return; + } + next(); + }, + ), +})); + +import teamRouter from './index'; + +const app = express(); +app.use(express.json()); +app.use('/api/teams', teamRouter); + +describe('API Key Routes', () => { + const teamId = 'team-1'; + + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE team_members ( + team_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (team_id, user_id), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + `); + + // Insert test users + const insertUser = testDb.prepare( + 'INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)', + ); + insertUser.run(adminUser.id, adminUser.email, adminUser.name, adminUser.role); + insertUser.run(leadUser.id, leadUser.email, leadUser.name, leadUser.role); + insertUser.run(memberUser.id, memberUser.email, memberUser.name, memberUser.role); + + // Insert team + testDb + .prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)') + .run(teamId, 'Test Team', 'TEST'); + + // Lead is a lead, member is a member + testDb + .prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)') + .run(teamId, leadUser.id, 'lead'); + testDb + .prepare('INSERT INTO team_members (team_id, user_id, role) VALUES (?, ?, ?)') + .run(teamId, memberUser.id, 'member'); + }); + + afterAll(() => { + testDb.close(); + }); + + beforeEach(() => { + currentUser = adminUser; + // Clean up api keys between tests + testDb.exec('DELETE FROM team_api_keys'); + testDb.exec('DELETE FROM audit_log'); + }); + + describe('POST /api/teams/:id/api-keys', () => { + it('should create an API key and return raw key once', async () => { + const res = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'My Key' }); + + expect(res.status).toBe(201); + expect(res.body.name).toBe('My Key'); + expect(res.body.rawKey).toMatch(/^dps_[0-9a-f]{32}$/); + expect(res.body.key_prefix).toMatch(/^dps_/); + expect(res.body.team_id).toBe(teamId); + expect(res.body.created_by).toBe(adminUser.id); + // key_hash should not be returned + expect(res.body.key_hash).toBeUndefined(); + }); + + it('should log an audit event on creation', async () => { + await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'Audit Key' }); + + const audit = testDb + .prepare("SELECT * FROM audit_log WHERE action = 'api_key.created'") + .get() as { action: string; resource_type: string; details: string } | undefined; + + expect(audit).toBeDefined(); + expect(audit!.resource_type).toBe('team_api_key'); + expect(JSON.parse(audit!.details).key_name).toBe('Audit Key'); + }); + + it('should return 400 when name is missing', async () => { + const res = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({}); + + expect(res.status).toBe(400); + }); + + it('should return 400 when name is empty', async () => { + const res = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: ' ' }); + + expect(res.status).toBe(400); + }); + + it('should allow team lead to create keys', async () => { + currentUser = leadUser; + + const res = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'Lead Key' }); + + expect(res.status).toBe(201); + }); + + it('should deny regular member from creating keys', async () => { + currentUser = memberUser; + + const res = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'Member Key' }); + + expect(res.status).toBe(403); + }); + }); + + describe('GET /api/teams/:id/api-keys', () => { + it('should list keys without raw key or key_hash', async () => { + // Create a key first + await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'List Key' }); + + const res = await request(app).get(`/api/teams/${teamId}/api-keys`); + + expect(res.status).toBe(200); + expect(res.body).toHaveLength(1); + expect(res.body[0].name).toBe('List Key'); + expect(res.body[0].key_prefix).toBeDefined(); + expect(res.body[0].rawKey).toBeUndefined(); + expect(res.body[0].key_hash).toBeUndefined(); + }); + + it('should return empty array when no keys exist', async () => { + const res = await request(app).get(`/api/teams/${teamId}/api-keys`); + + expect(res.status).toBe(200); + expect(res.body).toEqual([]); + }); + + it('should deny regular member from listing keys', async () => { + currentUser = memberUser; + + const res = await request(app).get(`/api/teams/${teamId}/api-keys`); + + expect(res.status).toBe(403); + }); + }); + + describe('DELETE /api/teams/:id/api-keys/:keyId', () => { + it('should revoke an API key', async () => { + const createRes = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'Delete Key' }); + + const keyId = createRes.body.id; + + const res = await request(app).delete( + `/api/teams/${teamId}/api-keys/${keyId}`, + ); + + expect(res.status).toBe(204); + + // Verify key is gone + const listRes = await request(app).get(`/api/teams/${teamId}/api-keys`); + expect(listRes.body).toHaveLength(0); + }); + + it('should log an audit event on revocation', async () => { + const createRes = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'Revoke Key' }); + + const keyId = createRes.body.id; + testDb.exec('DELETE FROM audit_log'); // Clear create event + + await request(app).delete(`/api/teams/${teamId}/api-keys/${keyId}`); + + const audit = testDb + .prepare("SELECT * FROM audit_log WHERE action = 'api_key.revoked'") + .get() as { action: string; resource_type: string; details: string } | undefined; + + expect(audit).toBeDefined(); + expect(audit!.resource_type).toBe('team_api_key'); + expect(JSON.parse(audit!.details).key_name).toBe('Revoke Key'); + }); + + it('should return 404 when key does not exist', async () => { + const res = await request(app).delete( + `/api/teams/${teamId}/api-keys/nonexistent`, + ); + + expect(res.status).toBe(404); + }); + + it('should deny regular member from revoking keys', async () => { + // Create as admin + const createRes = await request(app) + .post(`/api/teams/${teamId}/api-keys`) + .send({ name: 'Protected Key' }); + + currentUser = memberUser; + + const res = await request(app).delete( + `/api/teams/${teamId}/api-keys/${createRes.body.id}`, + ); + + expect(res.status).toBe(403); + }); + }); +}); diff --git a/server/src/routes/teams/apiKeys.ts b/server/src/routes/teams/apiKeys.ts new file mode 100644 index 0000000..b794b5b --- /dev/null +++ b/server/src/routes/teams/apiKeys.ts @@ -0,0 +1,108 @@ +import { Router, Request, Response } from 'express'; +import { requireTeamLead } from '../../auth'; +import { getStores } from '../../stores'; +import { sendErrorResponse, ValidationError } from '../../utils/errors'; + +const router = Router({ mergeParams: true }); + +/** + * GET /api/teams/:id/api-keys + * List API keys for a team (team lead/admin only). Never returns raw key. + */ +router.get('/', requireTeamLead, (req: Request, res: Response): void => { + try { + const teamId = req.params.id; + const stores = getStores(); + const keys = stores.teamApiKeys.findByTeamId(teamId); + + // Strip key_hash from response + const sanitized = keys.map(({ key_hash: _hash, ...rest }) => rest); + res.json(sanitized); + } catch (error) { + sendErrorResponse(res, error, 'listing API keys'); + } +}); + +/** + * POST /api/teams/:id/api-keys + * Create a new API key. Returns raw key once. + */ +router.post('/', requireTeamLead, (req: Request, res: Response): void => { + try { + const teamId = req.params.id; + const stores = getStores(); + const { name } = req.body; + + if (!name || typeof name !== 'string' || name.trim().length === 0) { + throw new ValidationError('Name is required', 'name'); + } + + const result = stores.teamApiKeys.create({ + team_id: teamId, + name: name.trim(), + created_by: req.user!.id, + }); + + // Audit log + stores.auditLog.create({ + user_id: req.user!.id, + action: 'api_key.created', + resource_type: 'team_api_key', + resource_id: result.id, + details: JSON.stringify({ + team_id: teamId, + key_name: result.name, + key_prefix: result.key_prefix, + }), + ip_address: req.ip || null, + }); + + // Return raw key once (strip key_hash) + const { key_hash: _hash, ...sanitized } = result; + res.status(201).json(sanitized); + } catch (error) { + sendErrorResponse(res, error, 'creating API key'); + } +}); + +/** + * DELETE /api/teams/:id/api-keys/:keyId + * Revoke an API key. + */ +router.delete('/:keyId', requireTeamLead, (req: Request, res: Response): void => { + try { + const teamId = req.params.id; + const keyId = req.params.keyId; + const stores = getStores(); + + // Verify key belongs to this team + const keys = stores.teamApiKeys.findByTeamId(teamId); + const key = keys.find((k) => k.id === keyId); + if (!key) { + res.status(404).json({ error: 'API key not found' }); + return; + } + + stores.teamApiKeys.delete(keyId); + + // Audit log + stores.auditLog.create({ + user_id: req.user!.id, + action: 'api_key.revoked', + resource_type: 'team_api_key', + resource_id: keyId, + details: JSON.stringify({ + team_id: teamId, + key_name: key.name, + key_prefix: key.key_prefix, + }), + ip_address: req.ip || null, + }); + + res.status(204).send(); + } catch (error) { + sendErrorResponse(res, error, 'revoking API key'); + } +}); + +export default router; diff --git a/server/src/routes/teams/index.ts b/server/src/routes/teams/index.ts index f544acb..6943fd2 100644 --- a/server/src/routes/teams/index.ts +++ b/server/src/routes/teams/index.ts @@ -8,6 +8,10 @@ import { deleteTeam } from './delete'; import { addMember } from './members/add'; import { updateMember } from './members/update'; import { removeMember } from './members/remove'; +import apiKeyRoutes from './apiKeys'; +import apiKeyRateLimitRoutes from './apiKeyRateLimit'; +import apiKeyUsageRoutes from './apiKeyUsage'; +import { getOtlpStats } from './otlpStats'; const router = Router(); @@ -23,4 +27,12 @@ router.post('/:id/members', requireAdmin, addMember); router.put('/:id/members/:userId', requireAdmin, updateMember); router.delete('/:id/members/:userId', requireAdmin, removeMember); +// OTLP stats - read-only, any authenticated team member +router.get('/:id/otlp-stats', getOtlpStats); + +// API key management - team lead/admin only (auth handled by apiKeys router) +router.use('/:id/api-keys', apiKeyRoutes); +router.use('/:id/api-keys', apiKeyRateLimitRoutes); +router.use('/:id/api-keys', apiKeyUsageRoutes); + export default router; diff --git a/server/src/routes/teams/otlpStats.test.ts b/server/src/routes/teams/otlpStats.test.ts new file mode 100644 index 0000000..7b6ee5e --- /dev/null +++ b/server/src/routes/teams/otlpStats.test.ts @@ -0,0 +1,373 @@ +import request from 'supertest'; +import express from 'express'; +import Database from 'better-sqlite3'; + +const testDb = new Database(':memory:'); + +jest.mock('../../db', () => ({ + db: testDb, + default: testDb, +})); + +jest.mock('../../auth', () => ({ + requireAuth: jest.fn((_req, _res, next) => next()), + requireAdmin: jest.fn((_req, _res, next) => next()), + requireTeamLead: jest.fn((_req, _res, next) => next()), +})); + +import teamRouter from './index'; + +const app = express(); +app.use(express.json()); +app.use('/api/teams', teamRouter); + +describe('OTLP Stats Routes', () => { + const teamId = 'team-otlp-1'; + + beforeAll(() => { + testDb.exec(` + CREATE TABLE users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + password_hash TEXT, + oidc_subject TEXT, + role TEXT NOT NULL DEFAULT 'user', + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE teams ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + key TEXT, + description TEXT, + contact TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE team_members ( + team_id TEXT NOT NULL, + user_id TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (team_id, user_id), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE, + FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + ); + + CREATE TABLE services ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + team_id TEXT NOT NULL, + health_endpoint TEXT NOT NULL, + metrics_endpoint TEXT, + schema_config TEXT, + poll_interval_ms INTEGER NOT NULL DEFAULT 30000, + is_active INTEGER NOT NULL DEFAULT 1, + is_external INTEGER NOT NULL DEFAULT 0, + description TEXT, + last_poll_success INTEGER, + last_poll_error TEXT, + poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + + CREATE TABLE dependencies ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + name TEXT NOT NULL, + canonical_name TEXT, + description TEXT, + impact TEXT, + type TEXT NOT NULL DEFAULT 'other', + healthy INTEGER, + health_state INTEGER, + health_code INTEGER, + latency_ms REAL, + contact TEXT, + contact_override TEXT, + impact_override TEXT, + check_details TEXT, + error TEXT, + error_message TEXT, + skipped INTEGER NOT NULL DEFAULT 0, + last_checked TEXT, + last_status_change TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE service_poll_history ( + id TEXT PRIMARY KEY, + service_id TEXT NOT NULL, + error TEXT, + recorded_at TEXT NOT NULL, + FOREIGN KEY (service_id) REFERENCES services(id) ON DELETE CASCADE + ); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT, + FOREIGN KEY (team_id) REFERENCES teams(id) ON DELETE CASCADE + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE audit_log ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + action TEXT NOT NULL, + resource_type TEXT NOT NULL, + resource_id TEXT, + details TEXT, + ip_address TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_by TEXT + ); + `); + + testDb + .prepare('INSERT INTO users (id, email, name, role) VALUES (?, ?, ?, ?)') + .run('user-1', 'admin@test.com', 'Admin', 'admin'); + + testDb + .prepare('INSERT INTO teams (id, name, key) VALUES (?, ?, ?)') + .run(teamId, 'OTLP Team', 'OTLP'); + }); + + afterAll(() => { + testDb.close(); + }); + + beforeEach(() => { + testDb.exec('DELETE FROM services'); + testDb.exec('DELETE FROM dependencies'); + testDb.exec('DELETE FROM service_poll_history'); + testDb.exec('DELETE FROM team_api_keys'); + }); + + describe('GET /api/teams/:id/otlp-stats', () => { + it('should return stats for OTLP services only', async () => { + // Insert 2 OTLP services and 1 non-OTLP + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('svc-otlp-1', 'OTLP Svc 1', teamId, 'push://otlp', 'otlp', 1, 1, '2026-03-15T10:00:00Z'); + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('svc-otlp-2', 'OTLP Svc 2', teamId, 'push://otlp', 'otlp', 1, null, '2026-03-15T09:00:00Z'); + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active) VALUES (?, ?, ?, ?, ?, ?)' + ).run('svc-default', 'Default Svc', teamId, 'http://localhost:8080/health', 'default', 1); + + // Add a dependency to first OTLP service + testDb.prepare( + 'INSERT INTO dependencies (id, service_id, name, type) VALUES (?, ?, ?, ?)' + ).run('dep-1', 'svc-otlp-1', 'postgres', 'database'); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.services).toHaveLength(2); + expect(res.body.services[0].name).toBe('OTLP Svc 1'); + expect(res.body.services[0].dependency_count).toBe(1); + expect(res.body.services[1].name).toBe('OTLP Svc 2'); + expect(res.body.services[1].dependency_count).toBe(0); + expect(res.body.summary.total_otlp_services).toBe(2); + expect(res.body.summary.active_services).toBe(2); + expect(res.body.summary.services_never_pushed).toBe(1); + }); + + it('should return empty when no OTLP services exist', async () => { + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active) VALUES (?, ?, ?, ?, ?, ?)' + ).run('svc-default-2', 'Default Svc', teamId, 'http://localhost/health', 'default', 1); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.services).toHaveLength(0); + expect(res.body.summary.total_otlp_services).toBe(0); + expect(res.body.summary.active_services).toBe(0); + }); + + it('should handle team with no services at all', async () => { + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.services).toEqual([]); + expect(res.body.apiKeys).toEqual([]); + expect(res.body.summary.total_otlp_services).toBe(0); + }); + + it('should include API keys without exposing key_hash', async () => { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_at) VALUES (?, ?, ?, ?, ?, ?)' + ).run('key-1', teamId, 'Prod Key', 'hash123', 'dps_abcd', '2026-03-15T00:00:00Z'); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.apiKeys).toHaveLength(1); + expect(res.body.apiKeys[0].name).toBe('Prod Key'); + expect(res.body.apiKeys[0].key_prefix).toBe('dps_abcd'); + expect(res.body.apiKeys[0].key_hash).toBeUndefined(); + }); + + it('should return error details for failing services', async () => { + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success, last_poll_error) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('svc-err', 'Failing Svc', teamId, 'push://otlp', 'otlp', 1, 0, 'Connection refused'); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.services[0].last_push_success).toBe(0); + expect(res.body.services[0].last_push_error).toBe('Connection refused'); + expect(res.body.summary.services_with_errors).toBe(1); + }); + + it('should parse poll_warnings JSON', async () => { + const warnings = JSON.stringify(['Missing metric: latency', 'Unknown label: env']); + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success, poll_warnings) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('svc-warn', 'Warning Svc', teamId, 'push://otlp', 'otlp', 1, 1, warnings); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.services[0].last_push_warnings).toEqual([ + 'Missing metric: latency', + 'Unknown label: env', + ]); + }); + + it('should include rate limit fields on API keys', async () => { + // Key with custom rate limit + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, rate_limit_admin_locked, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('key-custom', teamId, 'Custom Key', 'hash-custom', 'dps_cust', 5000, 0, '2026-03-15T00:00:00Z'); + + // Key with default (null) rate limit, admin-locked + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, rate_limit_rpm, rate_limit_admin_locked, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' + ).run('key-default', teamId, 'Default Key', 'hash-default', 'dps_dflt', null, 1, '2026-03-15T00:00:00Z'); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.apiKeys).toHaveLength(2); + + const customKey = res.body.apiKeys.find((k: { id: string }) => k.id === 'key-custom'); + expect(customKey.rate_limit_rpm).toBe(5000); + expect(customKey.rate_limit_is_custom).toBe(true); + expect(customKey.rate_limit_admin_locked).toBe(false); + + const defaultKey = res.body.apiKeys.find((k: { id: string }) => k.id === 'key-default'); + expect(defaultKey.rate_limit_rpm).toBe(150000); // system default + expect(defaultKey.rate_limit_is_custom).toBe(false); + expect(defaultKey.rate_limit_admin_locked).toBe(true); + }); + + it('should include usage summary fields on API keys', async () => { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_at) VALUES (?, ?, ?, ?, ?, ?)' + ).run('key-usage', teamId, 'Usage Key', 'hash-usage', 'dps_usg', '2026-03-15T00:00:00Z'); + + // Insert usage buckets within the summary windows + const now = new Date(); + const recentBucket = new Date(now.getTime() - 30 * 60 * 1000).toISOString().slice(0, 13) + ':00:00'; + const dayAgoBucket = new Date(now.getTime() - 12 * 60 * 60 * 1000).toISOString().slice(0, 13) + ':00:00'; + + testDb.prepare( + 'INSERT INTO api_key_usage_buckets (api_key_id, bucket_start, granularity, push_count, rejected_count) VALUES (?, ?, ?, ?, ?)' + ).run('key-usage', recentBucket, 'hour', 100, 5); + testDb.prepare( + 'INSERT INTO api_key_usage_buckets (api_key_id, bucket_start, granularity, push_count, rejected_count) VALUES (?, ?, ?, ?, ?)' + ).run('key-usage', dayAgoBucket, 'hour', 200, 10); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + const key = res.body.apiKeys.find((k: { id: string }) => k.id === 'key-usage'); + expect(key.usage_1h).toBeDefined(); + expect(key.usage_24h).toBeGreaterThanOrEqual(100); + expect(key.usage_7d).toBeGreaterThanOrEqual(300); + expect(key.rejected_24h).toBeGreaterThanOrEqual(5); + expect(key.rejected_7d).toBeGreaterThanOrEqual(15); + }); + + it('should return zero usage when no buckets exist', async () => { + testDb.prepare( + 'INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_at) VALUES (?, ?, ?, ?, ?, ?)' + ).run('key-empty', teamId, 'Empty Key', 'hash-empty', 'dps_empt', '2026-03-15T00:00:00Z'); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + const key = res.body.apiKeys.find((k: { id: string }) => k.id === 'key-empty'); + expect(key.usage_1h).toBe(0); + expect(key.usage_24h).toBe(0); + expect(key.usage_7d).toBe(0); + expect(key.rejected_24h).toBe(0); + expect(key.rejected_7d).toBe(0); + }); + + it('should count errors in last 24h', async () => { + testDb.prepare( + 'INSERT INTO services (id, name, team_id, health_endpoint, health_endpoint_format, is_active, last_poll_success) VALUES (?, ?, ?, ?, ?, ?, ?)' + ).run('svc-hist', 'History Svc', teamId, 'push://otlp', 'otlp', 1, 1); + + // Insert recent errors (within 24h) + testDb.prepare( + 'INSERT INTO service_poll_history (id, service_id, error, recorded_at) VALUES (?, ?, ?, ?)' + ).run('ph-1', 'svc-hist', 'timeout', new Date().toISOString()); + testDb.prepare( + 'INSERT INTO service_poll_history (id, service_id, error, recorded_at) VALUES (?, ?, ?, ?)' + ).run('ph-2', 'svc-hist', 'connection reset', new Date().toISOString()); + // Insert a success (null error) + testDb.prepare( + 'INSERT INTO service_poll_history (id, service_id, error, recorded_at) VALUES (?, ?, ?, ?)' + ).run('ph-3', 'svc-hist', null, new Date().toISOString()); + + const res = await request(app).get(`/api/teams/${teamId}/otlp-stats`); + + expect(res.status).toBe(200); + expect(res.body.services[0].errors_24h).toBe(2); + }); + }); +}); diff --git a/server/src/routes/teams/otlpStats.ts b/server/src/routes/teams/otlpStats.ts new file mode 100644 index 0000000..1adfaeb --- /dev/null +++ b/server/src/routes/teams/otlpStats.ts @@ -0,0 +1,81 @@ +import { Request, Response } from 'express'; +import { getStores } from '../../stores'; +import { sendErrorResponse } from '../../utils/errors'; + +export function getOtlpStats(req: Request, res: Response): void { + try { + const { id } = req.params; + const stores = getStores(); + + const allServices = stores.services.findByTeamId(id); + const otlpServices = allServices.filter(s => s.health_endpoint_format === 'otlp'); + const apiKeys = stores.teamApiKeys.findByTeamId(id); + + // Usage summaries for rate limit display + const keyIds = apiKeys.map(k => k.id); + const now = new Date().toISOString(); + const minus1h = new Date(Date.now() - 60 * 60 * 1000).toISOString(); + const minus24h = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); + const minus7d = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString(); + const summaries1h = stores.apiKeyUsage.getSummaryForKeys(keyIds, minus1h, now); + const summaries24h = stores.apiKeyUsage.getSummaryForKeys(keyIds, minus24h, now); + const summaries7d = stores.apiKeyUsage.getSummaryForKeys(keyIds, minus7d, now); + const defaultRpm = parseInt(process.env.OTLP_PER_KEY_RATE_LIMIT_RPM ?? '150000', 10); + + const services = otlpServices.map(s => { + const depCount = stores.dependencies.findByServiceId(s.id).length; + const errors24h = stores.servicePollHistory.getErrorCount24h(s.id); + + let parsedWarnings: string[] | null = null; + if (s.poll_warnings) { + try { + parsedWarnings = JSON.parse(s.poll_warnings); + } catch { + parsedWarnings = null; + } + } + + return { + id: s.id, + name: s.name, + is_active: s.is_active, + last_push_success: s.last_poll_success, + last_push_error: s.last_poll_error, + last_push_warnings: parsedWarnings, + last_push_at: s.updated_at, + dependency_count: depCount, + errors_24h: errors24h, + schema_config: s.schema_config, + }; + }); + + const summary = { + total_otlp_services: otlpServices.length, + active_services: otlpServices.filter(s => s.is_active).length, + services_with_errors: otlpServices.filter(s => s.last_poll_success === 0).length, + services_never_pushed: otlpServices.filter(s => s.last_poll_success === null).length, + }; + + res.json({ + services, + apiKeys: apiKeys.map(k => ({ + id: k.id, + name: k.name, + key_prefix: k.key_prefix, + last_used_at: k.last_used_at, + created_at: k.created_at, + rate_limit_rpm: k.rate_limit_rpm ?? defaultRpm, + rate_limit_is_custom: k.rate_limit_rpm !== null, + rate_limit_admin_locked: Boolean(k.rate_limit_admin_locked), + usage_1h: summaries1h.get(k.id)?.push_count ?? 0, + usage_24h: summaries24h.get(k.id)?.push_count ?? 0, + usage_7d: summaries7d.get(k.id)?.push_count ?? 0, + rejected_24h: summaries24h.get(k.id)?.rejected_count ?? 0, + rejected_7d: summaries7d.get(k.id)?.rejected_count ?? 0, + })), + summary, + }); + } catch (error) { + sendErrorResponse(res, error, 'getting OTLP stats'); + } +} diff --git a/server/src/routes/wallboard/wallboard.test.ts b/server/src/routes/wallboard/wallboard.test.ts index 5c754b5..422d5d0 100644 --- a/server/src/routes/wallboard/wallboard.test.ts +++ b/server/src/routes/wallboard/wallboard.test.ts @@ -88,6 +88,11 @@ describe('Wallboard API', () => { last_poll_success INTEGER, last_poll_error TEXT, poll_warnings TEXT, + manifest_key TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', + manifest_config_id TEXT, + manifest_last_synced_values TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')) ); @@ -110,6 +115,10 @@ describe('Wallboard API', () => { check_details TEXT, error TEXT, error_message TEXT, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, skipped INTEGER NOT NULL DEFAULT 0, last_checked TEXT, last_status_change TEXT, @@ -121,8 +130,10 @@ describe('Wallboard API', () => { CREATE TABLE dependency_canonical_overrides ( id TEXT PRIMARY KEY, canonical_name TEXT NOT NULL UNIQUE, + team_id TEXT, contact_override TEXT, impact_override TEXT, + manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')), updated_by TEXT @@ -133,6 +144,8 @@ describe('Wallboard API', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), UNIQUE (dependency_id, linked_service_id) diff --git a/server/src/services/alerts/AlertService.test.ts b/server/src/services/alerts/AlertService.test.ts index 4175633..2922141 100644 --- a/server/src/services/alerts/AlertService.test.ts +++ b/server/src/services/alerts/AlertService.test.ts @@ -58,6 +58,7 @@ const mockService: Service = { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2026-01-01T00:00:00Z', updated_at: '2026-01-01T00:00:00Z', }; diff --git a/server/src/services/graph/DependencyGraphBuilder.test.ts b/server/src/services/graph/DependencyGraphBuilder.test.ts index ae014dc..1351f37 100644 --- a/server/src/services/graph/DependencyGraphBuilder.test.ts +++ b/server/src/services/graph/DependencyGraphBuilder.test.ts @@ -20,6 +20,7 @@ describe('DependencyGraphBuilder', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', is_active: 1, is_external: 0, description: null, @@ -49,6 +50,10 @@ describe('DependencyGraphBuilder', () => { check_details: '{"query": "SELECT 1"}', error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: new Date().toISOString(), last_status_change: null, @@ -56,8 +61,10 @@ describe('DependencyGraphBuilder', () => { updated_at: new Date().toISOString(), service_name: 'test-service', target_service_id: targetServiceId, + association_id: targetServiceId ? `assoc-${serviceId}-${targetServiceId}` : null, association_type: 'api_call', avg_latency_24h: 45, + is_auto_suggested: null, }); beforeEach(() => { @@ -134,6 +141,37 @@ describe('DependencyGraphBuilder', () => { expect(graph.nodes[0].data.isExternal).toBe(true); }); + it('should compute discoveredDependencyCount from trace-discovered deps', () => { + const service = createService('svc-1', 'User Service'); + const deps = [ + createDependency('svc-1', 'svc-2'), + createDependency('svc-1', 'svc-3'), + createDependency('svc-1', 'svc-4'), + ]; + deps[0].id = 'dep-1'; + deps[1].id = 'dep-2'; + deps[2].id = 'dep-3'; + deps[0].discovery_source = 'manual'; + deps[1].discovery_source = 'otlp_trace'; + deps[2].discovery_source = 'otlp_trace'; + + builder.addServiceNode(service, deps); + const graph = builder.build(); + + expect(graph.nodes[0].data.discoveredDependencyCount).toBe(2); + }); + + it('should not include discoveredDependencyCount when zero', () => { + const service = createService('svc-1', 'User Service'); + const deps = [createDependency('svc-1', 'svc-2')]; + deps[0].discovery_source = 'manual'; + + builder.addServiceNode(service, deps); + const graph = builder.build(); + + expect(graph.nodes[0].data.discoveredDependencyCount).toBeUndefined(); + }); + it('should not set isExternal for tracked services', () => { const service = createService('svc-1', 'Tracked Service'); @@ -309,6 +347,77 @@ describe('DependencyGraphBuilder', () => { expect(graph.edges[0].data.dependencyName).toBe('postgres-primary'); }); + it('should include discoverySource and isAutoSuggested on edge data', () => { + const service1 = createService('svc-1', 'User Service'); + const service2 = createService('svc-2', 'Order Service'); + const dep = createDependency('svc-1', 'svc-2'); + dep.discovery_source = 'otlp_trace'; + dep.is_auto_suggested = 1; + + builder.addServiceNode(service1, []); + builder.addServiceNode(service2, []); + builder.addEdge(dep); + const graph = builder.build(); + + expect(graph.edges[0].data.discoverySource).toBe('otlp_trace'); + expect(graph.edges[0].data.isAutoSuggested).toBe(true); + }); + + it('should default discoverySource to manual and isAutoSuggested to false', () => { + const service1 = createService('svc-1', 'User Service'); + const service2 = createService('svc-2', 'Order Service'); + const dep = createDependency('svc-1', 'svc-2'); + + builder.addServiceNode(service1, []); + builder.addServiceNode(service2, []); + builder.addEdge(dep); + const graph = builder.build(); + + expect(graph.edges[0].data.discoverySource).toBe('manual'); + expect(graph.edges[0].data.isAutoSuggested).toBe(false); + }); + + it('should include associationId when association exists', () => { + const service1 = createService('svc-1', 'User Service'); + const service2 = createService('svc-2', 'Order Service'); + const dep = createDependency('svc-1', 'svc-2'); + + builder.addServiceNode(service1, []); + builder.addServiceNode(service2, []); + builder.addEdge(dep); + const graph = builder.build(); + + expect(graph.edges[0].data.associationId).toBe('assoc-svc-1-svc-2'); + }); + + it('should not include associationId when no association', () => { + const service1 = createService('svc-1', 'User Service'); + builder.addServiceNode(service1, []); + builder.addExternalNode('external-redis', { + name: 'Redis', + teamId: 'external', + teamName: 'External', + healthEndpoint: '', + isActive: true, + dependencyCount: 1, + healthyCount: 1, + unhealthyCount: 0, + skippedCount: 0, + lastPollSuccess: null, + lastPollError: null, + isExternal: true, + }); + builder.setExternalNodeMap(new Map([['redis', 'external-redis']])); + + const dep = createDependency('svc-1', null); + dep.name = 'Redis'; + dep.association_id = null; + builder.addEdge(dep); + + const graph = builder.build(); + expect(graph.edges[0].data.associationId).toBeUndefined(); + }); + it('should set canonicalName to null when canonical_name is null', () => { const service1 = createService('svc-1', 'User Service'); const service2 = createService('svc-2', 'Order Service'); diff --git a/server/src/services/graph/DependencyGraphBuilder.ts b/server/src/services/graph/DependencyGraphBuilder.ts index 3aeb8ae..493d0bf 100644 --- a/server/src/services/graph/DependencyGraphBuilder.ts +++ b/server/src/services/graph/DependencyGraphBuilder.ts @@ -41,6 +41,7 @@ export class DependencyGraphBuilder { const skippedCount = uniqueDeps.filter(d => d.skipped === 1).length; const healthyCount = uniqueDeps.filter(d => d.healthy === 1 && d.skipped !== 1).length; const unhealthyCount = uniqueDeps.filter(d => d.healthy === 0 && d.skipped !== 1).length; + const discoveredDependencyCount = uniqueDeps.filter(d => d.discovery_source === 'otlp_trace').length; this.nodes.push({ id: service.id, @@ -60,6 +61,7 @@ export class DependencyGraphBuilder { skippedCount, serviceType, ...(service.is_external === 1 && { isExternal: true }), + ...(discoveredDependencyCount > 0 && { discoveredDependencyCount }), }, }); @@ -206,6 +208,9 @@ export class DependencyGraphBuilder { impact: dep.impact, effectiveContact, ...(dep.skipped === 1 && { skipped: true }), + discoverySource: dep.discovery_source ?? 'manual', + isAutoSuggested: dep.is_auto_suggested === 1, + ...(dep.association_id && { associationId: dep.association_id }), }; } } diff --git a/server/src/services/graph/ExternalNodeBuilder.test.ts b/server/src/services/graph/ExternalNodeBuilder.test.ts index ae6a4e7..8fff8f3 100644 --- a/server/src/services/graph/ExternalNodeBuilder.test.ts +++ b/server/src/services/graph/ExternalNodeBuilder.test.ts @@ -26,6 +26,10 @@ function createDep( check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: '2024-01-01T00:00:00Z', last_status_change: null, @@ -33,8 +37,10 @@ function createDep( updated_at: '2024-01-01T00:00:00Z', service_name: 'Test Service', target_service_id: targetServiceId, + association_id: null, association_type: null, avg_latency_24h: null, + is_auto_suggested: null, }; } @@ -173,6 +179,84 @@ describe('ExternalNodeBuilder', () => { }); }); + describe('applyEnrichment', () => { + const baseNodeData = ExternalNodeBuilder.buildNodeData('Redis', [createDep('svc-1', 'Redis')]); + + it('should overlay display_name as name', () => { + const enrichment = { + id: 'enr-1', + canonical_name: 'redis', + display_name: 'Production Redis', + description: null, + impact: null, + contact: null, + service_type: null, + created_at: '2024-01-01T00:00:00Z', + updated_at: '2024-01-01T00:00:00Z', + updated_by: null, + }; + + const result = ExternalNodeBuilder.applyEnrichment(baseNodeData, enrichment); + expect(result.name).toBe('Production Redis'); + }); + + it('should overlay service_type as serviceType', () => { + const enrichment = { + id: 'enr-1', + canonical_name: 'redis', + display_name: null, + description: null, + impact: null, + contact: null, + service_type: 'cache', + created_at: '2024-01-01T00:00:00Z', + updated_at: '2024-01-01T00:00:00Z', + updated_by: null, + }; + + const result = ExternalNodeBuilder.applyEnrichment(baseNodeData, enrichment); + expect(result.serviceType).toBe('cache'); + }); + + it('should include enriched description, impact, and contact', () => { + const enrichment = { + id: 'enr-1', + canonical_name: 'redis', + display_name: null, + description: 'Shared cache cluster', + impact: 'Session data unavailable', + contact: '{"team":"platform"}', + service_type: null, + created_at: '2024-01-01T00:00:00Z', + updated_at: '2024-01-01T00:00:00Z', + updated_by: null, + }; + + const result = ExternalNodeBuilder.applyEnrichment(baseNodeData, enrichment); + expect(result.enrichedDescription).toBe('Shared cache cluster'); + expect(result.enrichedImpact).toBe('Session data unavailable'); + expect(result.enrichedContact).toBe('{"team":"platform"}'); + }); + + it('should not override name when display_name is null', () => { + const enrichment = { + id: 'enr-1', + canonical_name: 'redis', + display_name: null, + description: null, + impact: null, + contact: null, + service_type: null, + created_at: '2024-01-01T00:00:00Z', + updated_at: '2024-01-01T00:00:00Z', + updated_by: null, + }; + + const result = ExternalNodeBuilder.applyEnrichment(baseNodeData, enrichment); + expect(result.name).toBe('Redis'); + }); + }); + describe('buildNameToIdMap', () => { it('should map normalized names to external IDs', () => { const deps = [ diff --git a/server/src/services/graph/ExternalNodeBuilder.ts b/server/src/services/graph/ExternalNodeBuilder.ts index 3369f7f..ed39c2a 100644 --- a/server/src/services/graph/ExternalNodeBuilder.ts +++ b/server/src/services/graph/ExternalNodeBuilder.ts @@ -1,5 +1,5 @@ import { createHash } from 'crypto'; -import { DependencyType } from '../../db/types'; +import { DependencyType, ExternalNodeEnrichment } from '../../db/types'; import { DependencyWithTarget, ServiceNodeData } from './types'; interface ExternalGroup { @@ -107,6 +107,23 @@ export class ExternalNodeBuilder { }; } + /** + * Apply enrichment metadata from ExternalNodeEnrichmentStore to an external node. + */ + static applyEnrichment( + nodeData: ServiceNodeData, + enrichment: ExternalNodeEnrichment + ): ServiceNodeData { + return { + ...nodeData, + ...(enrichment.display_name && { name: enrichment.display_name }), + ...(enrichment.service_type && { serviceType: enrichment.service_type }), + ...(enrichment.description && { enrichedDescription: enrichment.description }), + ...(enrichment.impact && { enrichedImpact: enrichment.impact }), + ...(enrichment.contact && { enrichedContact: enrichment.contact }), + }; + } + /** * Build a name-to-ID map for edge resolution. */ diff --git a/server/src/services/graph/GraphService.test.ts b/server/src/services/graph/GraphService.test.ts index cf7075e..1420ee2 100644 --- a/server/src/services/graph/GraphService.test.ts +++ b/server/src/services/graph/GraphService.test.ts @@ -52,11 +52,19 @@ describe('GraphService', () => { delete: jest.fn(), }; + const mockExternalNodeEnrichmentStore = { + findAll: jest.fn().mockReturnValue([]), + findByCanonicalName: jest.fn(), + upsert: jest.fn(), + delete: jest.fn(), + }; + const mockStores = { services: mockServiceStore, dependencies: mockDependencyStore, teams: mockTeamStore, canonicalOverrides: mockCanonicalOverrideStore, + externalNodeEnrichment: mockExternalNodeEnrichmentStore, } as unknown as StoreRegistry; beforeEach(() => { @@ -80,6 +88,56 @@ describe('GraphService', () => { activeServicesOnly: true, }); }); + + it('should apply external node enrichment from store', () => { + const service = createService('svc-1', 'User Service'); + const dep = createDependency('svc-1', null, 'cache'); + dep.name = 'Redis'; + + mockServiceStore.findActiveWithTeam.mockReturnValue([service]); + mockDependencyStore.findAllWithAssociationsAndLatency.mockReturnValue([dep]); + mockExternalNodeEnrichmentStore.findAll.mockReturnValue([{ + id: 'enr-1', + canonical_name: 'Redis', + display_name: 'Production Redis Cluster', + description: 'Shared cache', + impact: 'Session data loss', + contact: '{"team":"platform"}', + service_type: 'cache', + created_at: '2024-01-01T00:00:00Z', + updated_at: '2024-01-01T00:00:00Z', + updated_by: null, + }]); + + const graphService = new GraphService(undefined, mockStores); + const result = graphService.getFullGraph(); + + const externalNode = result.nodes.find(n => n.data.isExternal); + expect(externalNode).toBeDefined(); + expect(externalNode!.data.name).toBe('Production Redis Cluster'); + expect(externalNode!.data.serviceType).toBe('cache'); + expect(externalNode!.data.enrichedDescription).toBe('Shared cache'); + expect(externalNode!.data.enrichedImpact).toBe('Session data loss'); + expect(externalNode!.data.enrichedContact).toBe('{"team":"platform"}'); + }); + + it('should include discovery source on graph edges', () => { + const service1 = createService('svc-1', 'User Service'); + const service2 = createService('svc-2', 'DB Service'); + const dep = createDependency('svc-1', 'svc-2', 'database'); + dep.discovery_source = 'otlp_trace'; + dep.is_auto_suggested = 1; + + mockServiceStore.findActiveWithTeam.mockReturnValue([service1, service2]); + mockDependencyStore.findAllWithAssociationsAndLatency.mockReturnValue([dep]); + + const graphService = new GraphService(undefined, mockStores); + const result = graphService.getFullGraph(); + + expect(result.edges).toHaveLength(1); + expect(result.edges[0].data.discoverySource).toBe('otlp_trace'); + expect(result.edges[0].data.isAutoSuggested).toBe(true); + }); }); describe('getTeamGraph', () => { @@ -428,6 +486,7 @@ function createService(id: string, name: string): ServiceWithTeam { is_active: 1, is_external: 0, description: null, + health_endpoint_format: 'default', created_at: '2024-01-01T00:00:00Z', updated_at: '2024-01-01T00:00:00Z', }; @@ -456,6 +515,10 @@ function createDependency( check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: '2024-01-01T00:00:00Z', last_status_change: null, @@ -463,7 +526,9 @@ function createDependency( updated_at: '2024-01-01T00:00:00Z', service_name: 'Test Service', target_service_id: targetServiceId, + association_id: null, association_type: 'api_call', avg_latency_24h: null, + is_auto_suggested: null, }; } diff --git a/server/src/services/graph/GraphService.ts b/server/src/services/graph/GraphService.ts index 2c18ed7..8dc19f8 100644 --- a/server/src/services/graph/GraphService.ts +++ b/server/src/services/graph/GraphService.ts @@ -4,6 +4,7 @@ import type { IDependencyStore, ITeamStore, ICanonicalOverrideStore, + IExternalNodeEnrichmentStore, } from '../../stores/interfaces'; import { DependencyCanonicalOverride } from '../../db/types'; import { ServiceWithTeam, DependencyWithTarget, GraphResponse } from './types'; @@ -22,6 +23,7 @@ export class GraphService { private dependencyStore: IDependencyStore; private teamStore: ITeamStore; private canonicalOverrideStore: ICanonicalOverrideStore; + private externalNodeEnrichmentStore: IExternalNodeEnrichmentStore; constructor(typeInferencer?: ServiceTypeInferencer, stores?: StoreRegistry) { const storeRegistry = stores || getStores(); @@ -30,6 +32,7 @@ export class GraphService { this.dependencyStore = storeRegistry.dependencies; this.teamStore = storeRegistry.teams; this.canonicalOverrideStore = storeRegistry.canonicalOverrides; + this.externalNodeEnrichmentStore = storeRegistry.externalNodeEnrichment; } /** @@ -197,8 +200,15 @@ export class GraphService { const groups = ExternalNodeBuilder.groupUnassociatedDeps(dependencies); if (groups.size === 0) return; - for (const [, group] of groups) { - const nodeData = ExternalNodeBuilder.buildNodeData(group.name, group.deps); + const enrichments = this.externalNodeEnrichmentStore.findAll(); + const enrichmentMap = new Map(enrichments.map(e => [e.canonical_name.toLowerCase().trim(), e])); + + for (const [normalizedName, group] of groups) { + let nodeData = ExternalNodeBuilder.buildNodeData(group.name, group.deps); + const enrichment = enrichmentMap.get(normalizedName); + if (enrichment) { + nodeData = ExternalNodeBuilder.applyEnrichment(nodeData, enrichment); + } builder.addExternalNode(group.id, nodeData); } diff --git a/server/src/services/graph/ServiceTypeInferencer.test.ts b/server/src/services/graph/ServiceTypeInferencer.test.ts index 961af31..b4476fe 100644 --- a/server/src/services/graph/ServiceTypeInferencer.test.ts +++ b/server/src/services/graph/ServiceTypeInferencer.test.ts @@ -26,6 +26,10 @@ describe('ServiceTypeInferencer', () => { check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: new Date().toISOString(), last_status_change: null, @@ -33,8 +37,10 @@ describe('ServiceTypeInferencer', () => { updated_at: new Date().toISOString(), service_name: 'test-service', target_service_id: targetServiceId, + association_id: null, association_type: null, avg_latency_24h: null, + is_auto_suggested: null, }); describe('compute', () => { diff --git a/server/src/services/graph/types.ts b/server/src/services/graph/types.ts index bd226ee..f193c9f 100644 --- a/server/src/services/graph/types.ts +++ b/server/src/services/graph/types.ts @@ -1,4 +1,4 @@ -import { Service, Dependency, DependencyType, AssociationType } from '../../db/types'; +import { Service, Dependency, DependencyType, AssociationType, DiscoverySource } from '../../db/types'; /** * Service with team name joined from teams table @@ -13,8 +13,10 @@ export interface ServiceWithTeam extends Service { export interface DependencyWithTarget extends Dependency { service_name: string; target_service_id: string | null; + association_id: string | null; association_type: string | null; avg_latency_24h: number | null; + is_auto_suggested: number | null; } /** @@ -39,6 +41,10 @@ export interface ServiceNodeData { skippedCount: number; serviceType?: DependencyType; isExternal?: boolean; + discoveredDependencyCount?: number; + enrichedDescription?: string | null; + enrichedImpact?: string | null; + enrichedContact?: string | null; } /** @@ -69,6 +75,9 @@ export interface GraphEdgeData { impact?: string | null; effectiveContact?: string | null; skipped?: boolean; + discoverySource?: DiscoverySource; + isAutoSuggested?: boolean; + associationId?: string | null; } /** diff --git a/server/src/services/manifest/ManifestDiffer.test.ts b/server/src/services/manifest/ManifestDiffer.test.ts index 5b666d0..759c9c2 100644 --- a/server/src/services/manifest/ManifestDiffer.test.ts +++ b/server/src/services/manifest/ManifestDiffer.test.ts @@ -40,6 +40,7 @@ function makeService( manifest_managed: 1, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2026-01-01T00:00:00.000Z', updated_at: '2026-01-01T00:00:00.000Z', ...overrides, @@ -460,6 +461,43 @@ describe('ManifestDiffer', () => { }); }); + // ========================================================================= + // health_endpoint_format drift detection + // ========================================================================= + describe('health_endpoint_format drift detection', () => { + it('detects health_endpoint_format changes', () => { + const entry = makeManifestEntry({ + health_endpoint_format: 'prometheus', + }); + const existing = makeService({ + health_endpoint_format: 'default', + manifest_last_synced_values: lastSynced({ + name: 'Service A', + health_endpoint: 'https://svc-a.example.com/health', + health_endpoint_format: 'default', + }), + }); + + const result = diffManifest([entry], [existing], makePolicy()); + + expect(result.toUpdate).toHaveLength(1); + expect(result.toUpdate[0].fields_changed).toContain('health_endpoint_format'); + }); + + it('treats matching health_endpoint_format as unchanged', () => { + const entry = makeManifestEntry({ + health_endpoint_format: 'prometheus', + }); + const existing = makeService({ + health_endpoint_format: 'prometheus', + }); + + const result = diffManifest([entry], [existing], makePolicy()); + + expect(result.unchanged).toEqual(['svc-id-1']); + }); + }); + // ========================================================================= // Complex scenario: multiple services, mixed outcomes // ========================================================================= diff --git a/server/src/services/manifest/ManifestDiffer.ts b/server/src/services/manifest/ManifestDiffer.ts index bf4a999..adbf754 100644 --- a/server/src/services/manifest/ManifestDiffer.ts +++ b/server/src/services/manifest/ManifestDiffer.ts @@ -16,6 +16,7 @@ const SYNCABLE_FIELDS = [ 'metrics_endpoint', 'poll_interval_ms', 'schema_config', + 'health_endpoint_format', ] as const; type SyncableField = (typeof SYNCABLE_FIELDS)[number]; @@ -236,6 +237,8 @@ function getManifestFieldValue( return entry.poll_interval_ms; case 'schema_config': return entry.schema_config; + case 'health_endpoint_format': + return entry.health_endpoint_format; } } @@ -259,6 +262,8 @@ function getDbFieldValue( return service.poll_interval_ms; case 'schema_config': return service.schema_config; + case 'health_endpoint_format': + return service.health_endpoint_format; } } diff --git a/server/src/services/manifest/ManifestSyncService.test.ts b/server/src/services/manifest/ManifestSyncService.test.ts index aba2b43..9fa0d7e 100644 --- a/server/src/services/manifest/ManifestSyncService.test.ts +++ b/server/src/services/manifest/ManifestSyncService.test.ts @@ -98,6 +98,7 @@ function makeService(overrides: Partial = {}): Service { manifest_managed: 1, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2026-01-01T00:00:00.000Z', updated_at: '2026-01-01T00:00:00.000Z', ...overrides, @@ -1160,4 +1161,121 @@ describe('ManifestSyncService', () => { expect(result.errors[0]).toContain('dependency or linked service was removed'); }); }); + + // ========================================================================= + // health_endpoint_format support + // ========================================================================= + describe('health_endpoint_format support', () => { + it('creates OTLP service with empty health_endpoint and poll_interval_ms=0', async () => { + const entry = { + key: 'otlp-svc', + name: 'OTLP Service', + health_endpoint: '', + health_endpoint_format: 'otlp' as const, + }; + stores.manifestConfig.findByTeamId.mockReturnValue([makeConfig()]); + stores.manifestConfig.findById.mockReturnValue(makeConfig()); + stores.services.findByTeamId.mockReturnValue([]); + mockFetch.mockResolvedValue({ + success: true, + data: { version: 1, services: [entry] }, + url: 'https://example.com/manifest.json', + }); + mockValidate.mockReturnValue({ + valid: true, version: 1, service_count: 1, valid_count: 1, errors: [], warnings: [], + }); + mockDiff.mockReturnValue({ + toCreate: [entry], toUpdate: [], toDrift: [], toKeepLocal: [], + unchanged: [], toDeactivate: [], toDelete: [], removalDrift: [], + }); + + const result = await service.syncTeam('team-1', 'manual', 'user-1'); + expect(result.status).toBe('success'); + expect(result.summary.services.created).toBe(1); + expect(stores.services.create).toHaveBeenCalledWith( + expect.objectContaining({ + health_endpoint: '', + poll_interval_ms: 0, + health_endpoint_format: 'otlp', + }), + ); + }); + + it('creates Prometheus service with format passed through', async () => { + const entry = { + key: 'prom-svc', + name: 'Prometheus Service', + health_endpoint: 'https://prom.example.com/metrics', + health_endpoint_format: 'prometheus' as const, + }; + stores.manifestConfig.findByTeamId.mockReturnValue([makeConfig()]); + stores.manifestConfig.findById.mockReturnValue(makeConfig()); + stores.services.findByTeamId.mockReturnValue([]); + mockFetch.mockResolvedValue({ + success: true, + data: { version: 1, services: [entry] }, + url: 'https://example.com/manifest.json', + }); + mockValidate.mockReturnValue({ + valid: true, version: 1, service_count: 1, valid_count: 1, errors: [], warnings: [], + }); + mockDiff.mockReturnValue({ + toCreate: [entry], toUpdate: [], toDrift: [], toKeepLocal: [], + unchanged: [], toDeactivate: [], toDelete: [], removalDrift: [], + }); + + const result = await service.syncTeam('team-1', 'manual', 'user-1'); + expect(result.status).toBe('success'); + expect(stores.services.create).toHaveBeenCalledWith( + expect.objectContaining({ + health_endpoint: 'https://prom.example.com/metrics', + health_endpoint_format: 'prometheus', + }), + ); + }); + + it('includes health_endpoint_format in synced values snapshot', async () => { + const entry = { + key: 'prom-svc', + name: 'Prometheus Service', + health_endpoint: 'https://prom.example.com/metrics', + health_endpoint_format: 'prometheus' as const, + }; + stores.manifestConfig.findByTeamId.mockReturnValue([makeConfig()]); + stores.manifestConfig.findById.mockReturnValue(makeConfig()); + stores.services.findByTeamId.mockReturnValue([]); + stores.services.create.mockReturnValue(makeService({ id: 'new-prom-id' })); + mockFetch.mockResolvedValue({ + success: true, + data: { version: 1, services: [entry] }, + url: 'https://example.com/manifest.json', + }); + mockValidate.mockReturnValue({ + valid: true, version: 1, service_count: 1, valid_count: 1, errors: [], warnings: [], + }); + mockDiff.mockReturnValue({ + toCreate: [entry], toUpdate: [], toDrift: [], toKeepLocal: [], + unchanged: [], toDeactivate: [], toDelete: [], removalDrift: [], + }); + + await service.syncTeam('team-1', 'manual', 'user-1'); + + // The setManifestColumns call uses db.prepare().run() which receives the synced values JSON + // Collect all run() calls from the db.prepare mock + const dbPrepare = stores.services.db.prepare; + const runMock = dbPrepare.mock.results[0].value.run; + const allCalls = runMock.mock.calls; + + // Find the call that includes the health_endpoint_format in the JSON snapshot + const syncedCall = allCalls.find((args: unknown[]) => + args.some((arg: unknown) => typeof arg === 'string' && arg.includes('health_endpoint_format')), + ); + expect(syncedCall).toBeDefined(); + const jsonArg = syncedCall.find((arg: unknown) => + typeof arg === 'string' && arg.includes('health_endpoint_format'), + ); + const snapshot = JSON.parse(jsonArg as string); + expect(snapshot.health_endpoint_format).toBe('prometheus'); + }); + }); }); diff --git a/server/src/services/manifest/ManifestSyncService.ts b/server/src/services/manifest/ManifestSyncService.ts index 1afa0dc..780470e 100644 --- a/server/src/services/manifest/ManifestSyncService.ts +++ b/server/src/services/manifest/ManifestSyncService.ts @@ -487,14 +487,16 @@ export class ManifestSyncService extends EventEmitter { continue; } + const isOtlp = entry.health_endpoint_format === 'otlp'; const service = txStores.services.create({ name: entry.name, team_id: teamId, - health_endpoint: entry.health_endpoint, + health_endpoint: isOtlp ? '' : entry.health_endpoint, metrics_endpoint: entry.metrics_endpoint ?? null, schema_config: entry.schema_config ? JSON.stringify(entry.schema_config) : null, - poll_interval_ms: entry.poll_interval_ms ?? 30000, + poll_interval_ms: isOtlp ? 0 : (entry.poll_interval_ms ?? 30000), description: entry.description ?? null, + health_endpoint_format: entry.health_endpoint_format ?? 'default', }); // Set manifest columns via raw update (not in ServiceUpdateInput) @@ -674,6 +676,7 @@ export class ManifestSyncService extends EventEmitter { if (entry.metrics_endpoint !== undefined) values.metrics_endpoint = entry.metrics_endpoint; if (entry.poll_interval_ms !== undefined) values.poll_interval_ms = entry.poll_interval_ms; if (entry.schema_config !== undefined) values.schema_config = entry.schema_config; + if (entry.health_endpoint_format !== undefined) values.health_endpoint_format = entry.health_endpoint_format; return values; } @@ -690,6 +693,8 @@ export class ManifestSyncService extends EventEmitter { case 'poll_interval_ms': return entry.poll_interval_ms; case 'schema_config': return entry.schema_config ? JSON.stringify(entry.schema_config) : null; + case 'health_endpoint_format': + return entry.health_endpoint_format ?? 'default'; default: return undefined; } } diff --git a/server/src/services/manifest/ManifestValidator.test.ts b/server/src/services/manifest/ManifestValidator.test.ts index 08972e2..81d667e 100644 --- a/server/src/services/manifest/ManifestValidator.test.ts +++ b/server/src/services/manifest/ManifestValidator.test.ts @@ -361,6 +361,120 @@ describe('ManifestValidator', () => { expect(result.valid).toBe(false); }); + // --- health_endpoint_format --- + it('accepts valid health_endpoint_format values', () => { + const formats = ['default', 'schema', 'prometheus', 'otlp']; + for (const format of formats) { + const svc = format === 'otlp' + ? validService({ health_endpoint_format: format, health_endpoint: undefined }) + : validService({ health_endpoint_format: format }); + const result = validateManifest({ version: 1, services: [svc] }); + expect(result.errors.filter(e => e.path.includes('health_endpoint_format'))).toHaveLength(0); + } + }); + + it('rejects invalid health_endpoint_format value', () => { + const result = validateManifest({ + version: 1, + services: [validService({ health_endpoint_format: 'xml' })], + }); + expect(result.valid).toBe(false); + expect(result.errors).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: 'services[0].health_endpoint_format', + message: expect.stringContaining('must be one of'), + }), + ]), + ); + }); + + it('allows empty health_endpoint for OTLP format', () => { + const result = validateManifest({ + version: 1, + services: [validService({ health_endpoint_format: 'otlp', health_endpoint: undefined })], + }); + expect(result.errors.filter(e => e.path.includes('health_endpoint'))).toHaveLength(0); + }); + + it('allows missing health_endpoint for OTLP format', () => { + const svc = { key: 'otlp-svc', name: 'OTLP Service', health_endpoint_format: 'otlp' }; + const result = validateManifest({ version: 1, services: [svc] }); + expect(result.errors.filter(e => e.path.includes('health_endpoint') && !e.path.includes('format'))).toHaveLength(0); + }); + + it('rejects non-zero poll_interval_ms for OTLP format', () => { + const result = validateManifest({ + version: 1, + services: [validService({ health_endpoint_format: 'otlp', health_endpoint: undefined, poll_interval_ms: 30000 })], + }); + expect(result.valid).toBe(false); + expect(result.errors).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: 'services[0].poll_interval_ms', + message: expect.stringContaining('OTLP'), + }), + ]), + ); + }); + + it('accepts poll_interval_ms of 0 for OTLP format', () => { + const result = validateManifest({ + version: 1, + services: [validService({ health_endpoint_format: 'otlp', health_endpoint: undefined, poll_interval_ms: 0 })], + }); + expect(result.errors.filter(e => e.path.includes('poll_interval_ms'))).toHaveLength(0); + }); + + it('validates schema_config as MetricSchemaConfig for OTLP format', () => { + const result = validateManifest({ + version: 1, + services: [validService({ + health_endpoint_format: 'otlp', + health_endpoint: undefined, + schema_config: { metrics: { up: 'healthy' }, labels: { service: 'name' } }, + })], + }); + expect(result.errors.filter(e => e.path.includes('schema_config'))).toHaveLength(0); + }); + + it('validates schema_config as MetricSchemaConfig for Prometheus format', () => { + const result = validateManifest({ + version: 1, + services: [validService({ + health_endpoint_format: 'prometheus', + schema_config: { metrics: { up: 'healthy' }, labels: { service: 'name' } }, + })], + }); + expect(result.errors.filter(e => e.path.includes('schema_config'))).toHaveLength(0); + }); + + it('validates schema_config as SchemaMapping for schema format (existing behavior)', () => { + const result = validateManifest({ + version: 1, + services: [validService({ + health_endpoint_format: 'schema', + schema_config: { root: 'deps', fields: { name: 'name', healthy: 'ok' } }, + })], + }); + expect(result.errors.filter(e => e.path.includes('schema_config'))).toHaveLength(0); + }); + + it('rejects SchemaMapping schema_config for Prometheus format', () => { + const result = validateManifest({ + version: 1, + services: [validService({ + health_endpoint_format: 'prometheus', + schema_config: { root: 'deps', fields: { name: 'name', healthy: 'ok' } }, + })], + }); + expect(result.valid).toBe(false); + expect(result.errors).toEqual( + expect.arrayContaining([expect.objectContaining({ path: 'services[0].schema_config' })]), + ); + }); + // --- unknown fields --- it('warns on unknown service fields', () => { const result = validateManifest({ diff --git a/server/src/services/manifest/ManifestValidator.ts b/server/src/services/manifest/ManifestValidator.ts index 153130f..de12ad9 100644 --- a/server/src/services/manifest/ManifestValidator.ts +++ b/server/src/services/manifest/ManifestValidator.ts @@ -7,13 +7,14 @@ import { isNonEmptyString, isNumber, validateSchemaConfig, + validateMetricSchemaConfig, VALID_ASSOCIATION_TYPES, MIN_POLL_INTERVAL_MS, MAX_POLL_INTERVAL_MS, TEAM_KEY_REGEX, MAX_KEY_LENGTH, } from '../../utils/validation'; -import type { AssociationType } from '../../db/types'; +import type { AssociationType, HealthEndpointFormat } from '../../db/types'; import { validateUrlHostname } from '../../utils/ssrf'; // --- Constants --- @@ -28,6 +29,8 @@ const KNOWN_TOP_LEVEL_KEYS = new Set([ 'associations', ]); +const VALID_FORMATS: HealthEndpointFormat[] = ['default', 'schema', 'prometheus', 'otlp']; + const KNOWN_SERVICE_FIELDS = new Set([ 'key', 'name', @@ -36,6 +39,7 @@ const KNOWN_SERVICE_FIELDS = new Set([ 'metrics_endpoint', 'poll_interval_ms', 'schema_config', + 'health_endpoint_format', ]); const KNOWN_ALIAS_FIELDS = new Set(['alias', 'canonical_name']); @@ -189,13 +193,38 @@ function validateServiceEntry( entryValid = false; } - // Required: health_endpoint (URL validation) - if (!isNonEmptyString(svc.health_endpoint)) { - addError(errors, `${path}.health_endpoint`, 'health_endpoint is required and must be a non-empty string'); - entryValid = false; + // Optional: health_endpoint_format (validate before health_endpoint so we know the format) + let format: HealthEndpointFormat = 'default'; + if (svc.health_endpoint_format !== undefined && svc.health_endpoint_format !== null) { + if (typeof svc.health_endpoint_format !== 'string' || + !VALID_FORMATS.includes(svc.health_endpoint_format as HealthEndpointFormat)) { + addError( + errors, + `${path}.health_endpoint_format`, + `health_endpoint_format must be one of: ${VALID_FORMATS.join(', ')}`, + ); + entryValid = false; + } else { + format = svc.health_endpoint_format as HealthEndpointFormat; + } + } + + // Required: health_endpoint (URL validation) — relaxed for OTLP (push-only, no endpoint needed) + if (format === 'otlp') { + // OTLP: health_endpoint is optional (may be empty or missing) + if (svc.health_endpoint !== undefined && svc.health_endpoint !== null && svc.health_endpoint !== '') { + if (!validateManifestUrl(svc.health_endpoint, `${path}.health_endpoint`, errors, warnings)) { + entryValid = false; + } + } } else { - if (!validateManifestUrl(svc.health_endpoint, `${path}.health_endpoint`, errors, warnings)) { + if (!isNonEmptyString(svc.health_endpoint)) { + addError(errors, `${path}.health_endpoint`, 'health_endpoint is required and must be a non-empty string'); entryValid = false; + } else { + if (!validateManifestUrl(svc.health_endpoint, `${path}.health_endpoint`, errors, warnings)) { + entryValid = false; + } } } @@ -217,8 +246,13 @@ function validateServiceEntry( } } - // Optional: poll_interval_ms (bounds check) - if (svc.poll_interval_ms !== undefined && svc.poll_interval_ms !== null) { + // Optional: poll_interval_ms (bounds check) — OTLP forces 0 + if (format === 'otlp') { + if (svc.poll_interval_ms !== undefined && svc.poll_interval_ms !== null && svc.poll_interval_ms !== 0) { + addError(errors, `${path}.poll_interval_ms`, 'poll_interval_ms must be 0 for OTLP format (push-only)'); + entryValid = false; + } + } else if (svc.poll_interval_ms !== undefined && svc.poll_interval_ms !== null) { if (!isNumber(svc.poll_interval_ms) || !Number.isInteger(svc.poll_interval_ms)) { addError(errors, `${path}.poll_interval_ms`, 'poll_interval_ms must be an integer'); entryValid = false; @@ -232,10 +266,14 @@ function validateServiceEntry( } } - // Optional: schema_config (validate via validateSchemaConfig) + // Optional: schema_config (format-aware validation) if (svc.schema_config !== undefined && svc.schema_config !== null) { try { - validateSchemaConfig(svc.schema_config); + if (format === 'prometheus' || format === 'otlp') { + validateMetricSchemaConfig(svc.schema_config); + } else { + validateSchemaConfig(svc.schema_config); + } } catch (err) { const message = err instanceof Error ? err.message : 'Invalid schema_config'; addError(errors, `${path}.schema_config`, message); diff --git a/server/src/services/manifest/types.test.ts b/server/src/services/manifest/types.test.ts index 6772416..f776b8e 100644 --- a/server/src/services/manifest/types.test.ts +++ b/server/src/services/manifest/types.test.ts @@ -565,6 +565,7 @@ describe('Updated existing types with manifest columns', () => { manifest_managed: 1, manifest_config_id: null, manifest_last_synced_values: JSON.stringify({ name: 'Test', health_endpoint: 'https://test.local/health' }), + health_endpoint_format: 'default', created_at: '2026-01-01T00:00:00Z', updated_at: '2026-01-01T00:00:00Z', }; @@ -592,6 +593,7 @@ describe('Updated existing types with manifest columns', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: '2026-01-01T00:00:00Z', updated_at: '2026-01-01T00:00:00Z', }; @@ -665,6 +667,8 @@ describe('Updated existing types with manifest columns', () => { dependency_id: 'dep-1', linked_service_id: 'svc-2', association_type: 'api_call', + is_auto_suggested: 0, + is_dismissed: 0, manifest_managed: 1, created_at: '2026-01-01T00:00:00Z', }; diff --git a/server/src/services/manifest/types.ts b/server/src/services/manifest/types.ts index fa5ef5d..3e557b4 100644 --- a/server/src/services/manifest/types.ts +++ b/server/src/services/manifest/types.ts @@ -1,4 +1,4 @@ -import { AssociationType } from '../../db/types'; +import { AssociationType, HealthEndpointFormat } from '../../db/types'; // --- DPS-49a: Sync policy types --- @@ -75,6 +75,7 @@ export interface ManifestServiceEntry { metrics_endpoint?: string; poll_interval_ms?: number; schema_config?: Record; + health_endpoint_format?: HealthEndpointFormat; } /** An alias entry in the manifest JSON. */ diff --git a/server/src/services/polling/AutoAssociator.test.ts b/server/src/services/polling/AutoAssociator.test.ts new file mode 100644 index 0000000..3e2e483 --- /dev/null +++ b/server/src/services/polling/AutoAssociator.test.ts @@ -0,0 +1,320 @@ +import { AutoAssociator } from './AutoAssociator'; +import { Service, ProactiveDepsStatus } from '../../db/types'; +import { StoreRegistry } from '../../stores'; + +/** Minimal Service factory */ +function makeService(overrides: Partial = {}): Service { + return { + id: 'svc-source', + name: 'source-service', + team_id: 'team-1', + health_endpoint: '', + health_endpoint_format: 'otlp', + poll_interval_ms: 0, + is_active: 1, + is_external: 0, + description: null, + metrics_endpoint: null, + schema_config: null, + last_poll_success: null, + last_poll_error: null, + poll_warnings: null, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + ...overrides, + } as Service; +} + +/** Minimal ProactiveDepsStatus factory */ +function makeDepsStatus(overrides: Partial = {}): ProactiveDepsStatus { + return { + name: 'target-service', + healthy: true, + health: { state: 0, code: 200, latency: 50 }, + lastChecked: new Date().toISOString(), + type: 'rest', + discovery_source: 'otlp_trace', + ...overrides, + } as ProactiveDepsStatus; +} + +/** Build mock StoreRegistry with spies */ +function createMockStores(overrides: Record = {}) { + return { + services: { + findByTeamId: jest.fn().mockReturnValue([]), + }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([]), + }, + associations: { + existsForDependencyAndService: jest.fn().mockReturnValue(false), + create: jest.fn().mockReturnValue({ id: 'assoc-new' }), + }, + aliases: { + resolveAlias: jest.fn().mockReturnValue(null), + }, + ...overrides, + } as unknown as StoreRegistry; +} + +describe('AutoAssociator', () => { + describe('processDiscoveredDependencies', () => { + it('creates auto-suggested association on exact name match (case-insensitive)', () => { + const targetSvc = makeService({ id: 'svc-target', name: 'Target-Service' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'target-service', canonical_name: null }, + ]), + }, + }); + + const associator = new AutoAssociator(stores); + const sourceService = makeService(); + const deps = [makeDepsStatus({ name: 'target-service', type: 'rest' })]; + + associator.processDiscoveredDependencies(sourceService, deps, 'team-1'); + + expect(stores.associations.create).toHaveBeenCalledWith({ + dependency_id: 'dep-1', + linked_service_id: 'svc-target', + association_type: 'api_call', + is_auto_suggested: true, + }); + }); + + it('creates association via canonical name match through alias resolution', () => { + const targetSvc = makeService({ id: 'svc-pg', name: 'PostgreSQL' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'postgres', canonical_name: null }, + ]), + }, + aliases: { + resolveAlias: jest.fn().mockReturnValue('PostgreSQL'), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies(makeService(), [makeDepsStatus({ name: 'postgres', type: 'database' })], 'team-1'); + + expect(stores.aliases.resolveAlias).toHaveBeenCalledWith('postgres'); + expect(stores.associations.create).toHaveBeenCalledWith( + expect.objectContaining({ + dependency_id: 'dep-1', + linked_service_id: 'svc-pg', + association_type: 'database', + is_auto_suggested: true, + }), + ); + }); + + it('uses canonical_name from dependency record when available', () => { + const targetSvc = makeService({ id: 'svc-pg', name: 'PostgreSQL' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'postgres', canonical_name: 'PostgreSQL' }, + ]), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies(makeService(), [makeDepsStatus({ name: 'postgres', type: 'database' })], 'team-1'); + + // Should NOT call resolveAlias since canonical_name is already set + expect(stores.aliases.resolveAlias).not.toHaveBeenCalled(); + expect(stores.associations.create).toHaveBeenCalledWith( + expect.objectContaining({ + linked_service_id: 'svc-pg', + }), + ); + }); + + it('skips self-links (source service === target service)', () => { + const sourceService = makeService({ id: 'svc-1', name: 'my-service' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([sourceService]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'my-service', canonical_name: null }, + ]), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies(sourceService, [makeDepsStatus({ name: 'my-service' })], 'team-1'); + + expect(stores.associations.create).not.toHaveBeenCalled(); + }); + + it('skips when association already exists (not duplicated)', () => { + const targetSvc = makeService({ id: 'svc-target', name: 'target-service' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'target-service', canonical_name: null }, + ]), + }, + associations: { + existsForDependencyAndService: jest.fn().mockReturnValue(true), + create: jest.fn(), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies(makeService(), [makeDepsStatus({ name: 'target-service' })], 'team-1'); + + expect(stores.associations.create).not.toHaveBeenCalled(); + }); + + it('skips dismissed associations (does not re-suggest)', () => { + // existsForDependencyAndService returns true for dismissed associations too + const targetSvc = makeService({ id: 'svc-target', name: 'target-service' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'target-service', canonical_name: null }, + ]), + }, + associations: { + existsForDependencyAndService: jest.fn().mockReturnValue(true), // dismissed still returns true + create: jest.fn(), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies(makeService(), [makeDepsStatus({ name: 'target-service' })], 'team-1'); + + expect(stores.associations.create).not.toHaveBeenCalled(); + }); + + it('maps dependency type to correct association_type', () => { + const teamServices = [ + makeService({ id: 'svc-db', name: 'postgres' }), + makeService({ id: 'svc-cache', name: 'redis' }), + makeService({ id: 'svc-mq', name: 'kafka' }), + makeService({ id: 'svc-grpc', name: 'grpc-service' }), + makeService({ id: 'svc-rest', name: 'rest-api' }), + makeService({ id: 'svc-unknown', name: 'unknown-svc' }), + ]; + + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue(teamServices) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-db', name: 'postgres', canonical_name: null }, + { id: 'dep-cache', name: 'redis', canonical_name: null }, + { id: 'dep-mq', name: 'kafka', canonical_name: null }, + { id: 'dep-grpc', name: 'grpc-service', canonical_name: null }, + { id: 'dep-rest', name: 'rest-api', canonical_name: null }, + { id: 'dep-unknown', name: 'unknown-svc', canonical_name: null }, + ]), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies(makeService(), [ + makeDepsStatus({ name: 'postgres', type: 'database' }), + makeDepsStatus({ name: 'redis', type: 'cache' }), + makeDepsStatus({ name: 'kafka', type: 'message_queue' }), + makeDepsStatus({ name: 'grpc-service', type: 'grpc' }), + makeDepsStatus({ name: 'rest-api', type: 'rest' }), + makeDepsStatus({ name: 'unknown-svc', type: 'custom' }), + ], 'team-1'); + + const calls = (stores.associations.create as jest.Mock).mock.calls; + expect(calls).toHaveLength(6); + + const byDep = Object.fromEntries( + calls.map((c: Array<{ dependency_id: string; association_type: string }>) => [c[0].dependency_id, c[0].association_type]), + ); + expect(byDep['dep-db']).toBe('database'); + expect(byDep['dep-cache']).toBe('cache'); + expect(byDep['dep-mq']).toBe('message_queue'); + expect(byDep['dep-grpc']).toBe('api_call'); + expect(byDep['dep-rest']).toBe('api_call'); + expect(byDep['dep-unknown']).toBe('other'); + }); + + it('catches UNIQUE constraint violation as no-op', () => { + const targetSvc = makeService({ id: 'svc-target', name: 'target-service' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'target-service', canonical_name: null }, + ]), + }, + associations: { + existsForDependencyAndService: jest.fn().mockReturnValue(false), + create: jest.fn().mockImplementation(() => { + throw new Error('UNIQUE constraint failed: dependency_associations.dependency_id, dependency_associations.linked_service_id'); + }), + }, + }); + + const associator = new AutoAssociator(stores); + + // Should not throw + expect(() => { + associator.processDiscoveredDependencies(makeService(), [makeDepsStatus({ name: 'target-service' })], 'team-1'); + }).not.toThrow(); + }); + + it('does nothing when no dependencies provided', () => { + const stores = createMockStores(); + const associator = new AutoAssociator(stores); + + associator.processDiscoveredDependencies(makeService(), [], 'team-1'); + + expect(stores.services.findByTeamId).not.toHaveBeenCalled(); + }); + + it('skips dependencies with no matching registered service', () => { + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([]) }, // no team services + dependencies: { + findByServiceId: jest.fn().mockReturnValue([ + { id: 'dep-1', name: 'unregistered-target', canonical_name: null }, + ]), + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies( + makeService(), + [makeDepsStatus({ name: 'unregistered-target' })], + 'team-1', + ); + + expect(stores.associations.create).not.toHaveBeenCalled(); + }); + + it('skips when dependency record not found in DB', () => { + const targetSvc = makeService({ id: 'svc-target', name: 'target-service' }); + const stores = createMockStores({ + services: { findByTeamId: jest.fn().mockReturnValue([targetSvc]) }, + dependencies: { + findByServiceId: jest.fn().mockReturnValue([]), // no deps in DB yet + }, + }); + + const associator = new AutoAssociator(stores); + associator.processDiscoveredDependencies( + makeService(), + [makeDepsStatus({ name: 'target-service' })], + 'team-1', + ); + + expect(stores.associations.create).not.toHaveBeenCalled(); + }); + }); +}); diff --git a/server/src/services/polling/AutoAssociator.ts b/server/src/services/polling/AutoAssociator.ts new file mode 100644 index 0000000..150d711 --- /dev/null +++ b/server/src/services/polling/AutoAssociator.ts @@ -0,0 +1,125 @@ +import { Service, AssociationType, ProactiveDepsStatus } from '../../db/types'; +import { StoreRegistry } from '../../stores'; +import logger from '../../utils/logger'; + +/** + * Automatically creates dependency_associations between trace-discovered + * dependencies and registered services when a match is found. + * + * Matching strategy (exact only, no fuzzy): + * 1. Case-insensitive name match against team services + * 2. Canonical name resolution via DependencyAliasStore → match service name + * + * Skips self-links, already-associated pairs (including dismissed), and + * catches UNIQUE constraint violations as no-ops for race-condition safety. + */ +export class AutoAssociator { + constructor(private stores: StoreRegistry) {} + + /** + * For each trace-discovered dependency owned by sourceService, attempt to + * find a matching registered service and create an auto-suggested association. + */ + processDiscoveredDependencies( + sourceService: Service, + dependencies: ProactiveDepsStatus[], + teamId: string, + ): void { + if (dependencies.length === 0) return; + + const teamServices = this.stores.services.findByTeamId(teamId); + const sourceDeps = this.stores.dependencies.findByServiceId(sourceService.id); + + for (const dep of dependencies) { + try { + // Look up the persisted dependency record by name + const depRecord = sourceDeps.find( + (d) => d.name.toLowerCase() === dep.name.toLowerCase(), + ); + if (!depRecord) continue; + + // Find a matching registered service + const targetService = this.findTargetService( + dep.name, + teamServices, + depRecord.canonical_name, + ); + if (!targetService) continue; + + // Skip self-links + if (targetService.id === sourceService.id) continue; + + // Skip if any association already exists (including dismissed) + if (this.stores.associations.existsForDependencyAndService(depRecord.id, targetService.id)) { + continue; + } + + const associationType = this.mapDependencyTypeToAssociationType(dep.type); + + this.stores.associations.create({ + dependency_id: depRecord.id, + linked_service_id: targetService.id, + association_type: associationType, + is_auto_suggested: true, + }); + } catch (err) { + // Catch UNIQUE constraint violations as no-ops (race condition safety) + const message = err instanceof Error ? err.message : String(err); + if (message.includes('UNIQUE constraint failed')) { + continue; + } + logger.warn({ err, depName: dep.name }, 'AutoAssociator: failed to process dependency'); + } + } + } + + /** + * Find a registered service matching the dependency target name. + * 1. Exact case-insensitive match on service name + * 2. Resolve canonical name via alias store, then match service name + */ + private findTargetService( + depName: string, + teamServices: Service[], + canonicalName: string | null, + ): Service | undefined { + const depNameLower = depName.toLowerCase(); + + // 1. Exact name match (case-insensitive) + const exactMatch = teamServices.find( + (s) => s.name.toLowerCase() === depNameLower, + ); + if (exactMatch) return exactMatch; + + // 2. Canonical name match via alias resolution + const resolvedCanonical = canonicalName ?? this.stores.aliases.resolveAlias(depName); + if (resolvedCanonical) { + const canonicalLower = resolvedCanonical.toLowerCase(); + const aliasMatch = teamServices.find( + (s) => s.name.toLowerCase() === canonicalLower, + ); + if (aliasMatch) return aliasMatch; + } + + return undefined; + } + + /** + * Map dependency type strings to association type enum values. + */ + private mapDependencyTypeToAssociationType(type?: string): AssociationType { + switch (type) { + case 'database': + return 'database'; + case 'cache': + return 'cache'; + case 'message_queue': + return 'message_queue'; + case 'rest': + case 'grpc': + return 'api_call'; + default: + return 'other'; + } + } +} diff --git a/server/src/services/polling/DependencyParser.test.ts b/server/src/services/polling/DependencyParser.test.ts index baf3cdb..4f442f2 100644 --- a/server/src/services/polling/DependencyParser.test.ts +++ b/server/src/services/polling/DependencyParser.test.ts @@ -349,4 +349,83 @@ describe('DependencyParser', () => { expect(() => parser.parse(data)).toThrow('missing healthy'); }); }); + + describe('format-aware dispatch', () => { + it('should use default array parser when format is "default"', () => { + const parser = new DependencyParser(); + const result = parser.parse([{ name: 'test', healthy: true }], null, undefined, 'default'); + expect(result).toHaveLength(1); + expect(result[0].name).toBe('test'); + }); + + it('should use SchemaMapper when format is "schema" with schemaConfig', () => { + const parser = new DependencyParser(); + const schema: SchemaMapping = { + root: 'checks', + fields: { + name: 'checkName', + healthy: { field: 'status', equals: 'ok' }, + }, + }; + const data = { checks: [{ checkName: 'db', status: 'ok' }] }; + const result = parser.parse(data, schema, undefined, 'schema'); + expect(result).toHaveLength(1); + expect(result[0].name).toBe('db'); + expect(result[0].healthy).toBe(true); + }); + + it('should delegate to PrometheusParser when format is "prometheus"', () => { + const parser = new DependencyParser(); + const promText = [ + 'dependency_health_status{name="postgres"} 0', + 'dependency_health_healthy{name="postgres"} 1', + 'dependency_health_latency_ms{name="postgres"} 25', + ].join('\n'); + + const result = parser.parse(promText, null, undefined, 'prometheus'); + expect(result).toHaveLength(1); + expect(result[0].name).toBe('postgres'); + expect(result[0].healthy).toBe(true); + expect(result[0].health.latency).toBe(25); + }); + + it('should throw when format is "otlp"', () => { + const parser = new DependencyParser(); + expect(() => parser.parse({}, null, undefined, 'otlp')).toThrow( + 'OTLP services are push-only and cannot be polled' + ); + }); + + it('should default to "default" format when format is undefined', () => { + const parser = new DependencyParser(); + const result = parser.parse([{ name: 'test', healthy: true }]); + expect(result).toHaveLength(1); + expect(result[0].name).toBe('test'); + }); + + it('should aggregate warnings from PrometheusParser', () => { + const parser = new DependencyParser(); + // Line with a known metric but missing "name" label + const promText = 'dependency_health_status{} 0\n'; + + parser.parse(promText, null, undefined, 'prometheus'); + expect(parser.lastWarnings.length).toBeGreaterThan(0); + expect(parser.lastWarnings[0]).toMatch(/missing required "name" label/); + }); + + it('should aggregate warnings from SchemaMapper', () => { + const parser = new DependencyParser(); + const schema: SchemaMapping = { + root: 'checks', + fields: { + name: 'checkName', + healthy: { field: 'status', equals: 'ok' }, + }, + }; + // Non-object entry in array will be skipped with a warning + const data = { checks: [null, { checkName: 'db', status: 'ok' }] }; + parser.parse(data, schema, undefined, 'schema'); + expect(parser.lastWarnings.length).toBeGreaterThan(0); + }); + }); }); diff --git a/server/src/services/polling/DependencyParser.ts b/server/src/services/polling/DependencyParser.ts index 42b4f03..b596f24 100644 --- a/server/src/services/polling/DependencyParser.ts +++ b/server/src/services/polling/DependencyParser.ts @@ -1,15 +1,18 @@ -import { ProactiveDepsStatus, DependencyType, SchemaMapping } from '../../db/types'; +import { ProactiveDepsStatus, DependencyType, SchemaMapping, HealthEndpointFormat, MetricSchemaConfig } from '../../db/types'; import { SchemaMapper } from './SchemaMapper'; +import { PrometheusParser } from './PrometheusParser'; +import { isMetricSchemaConfig } from './metricSchemaUtils'; /** * Parses health endpoint responses into ProactiveDepsStatus objects. * Handles both nested and flat response formats. * When a SchemaMapping is provided, delegates to SchemaMapper for custom schemas. + * Dispatches to format-specific parsers based on HealthEndpointFormat. */ export class DependencyParser { private _lastWarnings: string[] = []; - /** Warnings from the most recent parse() call (schema mapping only). */ + /** Warnings from the most recent parse() call. */ get lastWarnings(): string[] { return this._lastWarnings; } @@ -18,19 +21,37 @@ export class DependencyParser { * Parse a health endpoint response into an array of dependency statuses. * @param data - The raw response data (expected to be an array, or object for custom schema) * @param schemaConfig - Optional schema mapping for custom health endpoint formats + * @param serviceName - Optional service name for schema mapping context + * @param format - The health endpoint format to use for parsing (default: 'default') * @returns Array of parsed ProactiveDepsStatus objects - * @throws Error if the data format is invalid + * @throws Error if the data format is invalid or format is 'otlp' (push-only) */ - parse(data: unknown, schemaConfig?: SchemaMapping | null, serviceName?: string): ProactiveDepsStatus[] { + parse(data: unknown, schemaConfig?: SchemaMapping | MetricSchemaConfig | null, serviceName?: string, format?: HealthEndpointFormat): ProactiveDepsStatus[] { this._lastWarnings = []; + const effectiveFormat = format ?? 'default'; - if (schemaConfig) { - const mapper = new SchemaMapper(schemaConfig, serviceName); - const results = mapper.parse(data); - this._lastWarnings = mapper.warnings; + if (effectiveFormat === 'otlp') { + throw new Error('OTLP services are push-only and cannot be polled'); + } + + if (effectiveFormat === 'prometheus') { + const metricConfig = isMetricSchemaConfig(schemaConfig) ? schemaConfig : undefined; + const prometheusParser = new PrometheusParser(); + const results = prometheusParser.parse(data as string, metricConfig); + this._lastWarnings = prometheusParser.lastWarnings; return results; } + // 'schema' format or 'default' with schemaConfig + if (effectiveFormat === 'schema' || schemaConfig) { + if (schemaConfig) { + const mapper = new SchemaMapper(schemaConfig as SchemaMapping, serviceName); + const results = mapper.parse(data); + this._lastWarnings = mapper.warnings; + return results; + } + } + if (!Array.isArray(data)) { throw new Error('Invalid response: expected array'); } diff --git a/server/src/services/polling/DependencyUpsertService.test.ts b/server/src/services/polling/DependencyUpsertService.test.ts index 1795a5c..6c8a799 100644 --- a/server/src/services/polling/DependencyUpsertService.test.ts +++ b/server/src/services/polling/DependencyUpsertService.test.ts @@ -65,6 +65,10 @@ describe('DependencyUpsertService', () => { error TEXT, error_message TEXT, skipped INTEGER NOT NULL DEFAULT 0, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, last_checked TEXT, last_status_change TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), @@ -77,6 +81,8 @@ describe('DependencyUpsertService', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), UNIQUE (dependency_id, linked_service_id) @@ -93,6 +99,13 @@ describe('DependencyUpsertService', () => { id TEXT PRIMARY KEY, dependency_id TEXT NOT NULL, latency_ms INTEGER NOT NULL, + p50_ms REAL, + p95_ms REAL, + p99_ms REAL, + min_ms REAL, + max_ms REAL, + request_count INTEGER, + source TEXT NOT NULL DEFAULT 'poll', recorded_at TEXT NOT NULL ); @@ -141,6 +154,7 @@ describe('DependencyUpsertService', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: new Date().toISOString(), updated_at: new Date().toISOString(), }); @@ -353,4 +367,71 @@ describe('DependencyUpsertService', () => { expect(stored[0].contact).toBeNull(); }); }); + + describe('percentile recording', () => { + it('calls recordWithPercentiles when dep has percentiles', () => { + const upsertService = new DependencyUpsertService(mockErrorRecorder, stores); + const service = createService(); + + const dep = createDepStatus({ + name: 'HistogramDep', + health: { + state: 0, + code: 200, + latency: 25, + percentiles: { + p50: 10, + p95: 40, + p99: 80, + min: 2, + max: 100, + requestCount: 500, + }, + }, + }); + + upsertService.upsert(service, [dep]); + + // Verify percentile data was stored + const rows = testDb + .prepare('SELECT * FROM dependency_latency_history WHERE dependency_id = (SELECT id FROM dependencies WHERE name = ?)') + .all('HistogramDep') as Array>; + + expect(rows).toHaveLength(1); + expect(rows[0].latency_ms).toBe(25); + expect(rows[0].p50_ms).toBe(10); + expect(rows[0].p95_ms).toBe(40); + expect(rows[0].p99_ms).toBe(80); + expect(rows[0].min_ms).toBe(2); + expect(rows[0].max_ms).toBe(100); + expect(rows[0].request_count).toBe(500); + expect(rows[0].source).toBe('otlp_histogram'); + }); + + it('uses basic record() when dep has no percentiles', () => { + const upsertService = new DependencyUpsertService(mockErrorRecorder, stores); + const service = createService(); + + const dep = createDepStatus({ + name: 'PlainDep', + health: { + state: 0, + code: 200, + latency: 15, + }, + }); + + upsertService.upsert(service, [dep]); + + const rows = testDb + .prepare('SELECT * FROM dependency_latency_history WHERE dependency_id = (SELECT id FROM dependencies WHERE name = ?)') + .all('PlainDep') as Array>; + + expect(rows).toHaveLength(1); + expect(rows[0].latency_ms).toBe(15); + // No percentile columns should be populated + expect(rows[0].p50_ms).toBeNull(); + expect(rows[0].source).toBe('poll'); + }); + }); }); diff --git a/server/src/services/polling/DependencyUpsertService.ts b/server/src/services/polling/DependencyUpsertService.ts index eeb27bf..e687237 100644 --- a/server/src/services/polling/DependencyUpsertService.ts +++ b/server/src/services/polling/DependencyUpsertService.ts @@ -79,7 +79,15 @@ export class DependencyUpsertService { ); // Record latency history if latency is available - if (dep.health.latency > 0) { + if (dep.health.percentiles) { + this.latencyStore.recordWithPercentiles( + result.dependency.id, + dep.health.latency, + dep.health.percentiles, + now, + 'otlp_histogram', + ); + } else if (dep.health.latency > 0) { this.latencyStore.record(result.dependency.id, dep.health.latency, now); } } diff --git a/server/src/services/polling/HealthPollingService.test.ts b/server/src/services/polling/HealthPollingService.test.ts index 6823644..361152c 100644 --- a/server/src/services/polling/HealthPollingService.test.ts +++ b/server/src/services/polling/HealthPollingService.test.ts @@ -36,6 +36,7 @@ const createService = ( manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: new Date().toISOString(), updated_at: new Date().toISOString(), ...overrides, @@ -1082,3 +1083,85 @@ describe('HealthPollingService - host rate limiting', () => { expect(pollDeduplicator.size).toBe(0); }); }); + +describe('HealthPollingService - OTLP service skip', () => { + afterEach(async () => { + await HealthPollingService.resetInstance(); + }); + + it('should not add OTLP service to polling via addServiceToPolling', () => { + const otlpService = createService('svc-otlp', 'otlp-service', { + health_endpoint_format: 'otlp', + health_endpoint: '', + }); + const { stateManager, pollers, syncServices } = createPollingService([otlpService]); + + syncServices(); + + expect(stateManager.hasService('svc-otlp')).toBe(false); + expect(pollers.has('svc-otlp')).toBe(false); + }); + + it('should not add OTLP service via startService', () => { + const otlpService = createService('svc-otlp', 'otlp-service', { + health_endpoint_format: 'otlp', + health_endpoint: '', + }); + const { instance, stateManager, mockServiceStore } = createPollingService([]); + + mockServiceStore.findById.mockReturnValue(otlpService); + + instance.startService('svc-otlp'); + + expect(stateManager.hasService('svc-otlp')).toBe(false); + expect(mockLogger.info).toHaveBeenCalledWith( + { serviceId: 'svc-otlp', serviceName: 'otlp-service' }, + 'skipping OTLP service (push-only)' + ); + }); + + it('should not affect non-OTLP services', () => { + const defaultService = createService('svc-default', 'default-service', { + health_endpoint_format: 'default', + }); + const promService = createService('svc-prom', 'prom-service', { + health_endpoint_format: 'prometheus', + }); + const otlpService = createService('svc-otlp', 'otlp-service', { + health_endpoint_format: 'otlp', + health_endpoint: '', + }); + + const { stateManager, pollers, syncServices } = createPollingService([ + defaultService, + promService, + otlpService, + ]); + + syncServices(); + + expect(stateManager.hasService('svc-default')).toBe(true); + expect(stateManager.hasService('svc-prom')).toBe(true); + expect(stateManager.hasService('svc-otlp')).toBe(false); + expect(pollers.has('svc-default')).toBe(true); + expect(pollers.has('svc-prom')).toBe(true); + expect(pollers.has('svc-otlp')).toBe(false); + }); + + it('should not start OTLP service via startAll', async () => { + const otlpService = createService('svc-otlp', 'otlp-service', { + health_endpoint_format: 'otlp', + health_endpoint: '', + }); + const normalService = createService('svc-1', 'normal-service'); + + const { instance, stateManager } = createPollingService([otlpService, normalService]); + + instance.startAll(); + + expect(stateManager.hasService('svc-1')).toBe(true); + expect(stateManager.hasService('svc-otlp')).toBe(false); + + await instance.shutdown(); + }); +}); diff --git a/server/src/services/polling/HealthPollingService.ts b/server/src/services/polling/HealthPollingService.ts index 6bd893e..1cd93e9 100644 --- a/server/src/services/polling/HealthPollingService.ts +++ b/server/src/services/polling/HealthPollingService.ts @@ -82,6 +82,12 @@ export class HealthPollingService extends EventEmitter { return; } + // OTLP services are push-only — never poll them + if (service.health_endpoint_format === 'otlp') { + logger.info({ serviceId, serviceName: service.name }, 'skipping OTLP service (push-only)'); + return; + } + this.addServiceToPolling(service); logger.info({ serviceId, serviceName: service.name }, 'started polling service'); @@ -304,6 +310,11 @@ export class HealthPollingService extends EventEmitter { } private addServiceToPolling(service: Service): void { + // OTLP services are push-only — never poll them + if (service.health_endpoint_format === 'otlp') { + return; + } + // Add to state manager this.stateManager.addService(service); diff --git a/server/src/services/polling/OtlpParser.test.ts b/server/src/services/polling/OtlpParser.test.ts new file mode 100644 index 0000000..01e29c0 --- /dev/null +++ b/server/src/services/polling/OtlpParser.test.ts @@ -0,0 +1,939 @@ +import { OtlpParser } from './OtlpParser'; +import { OtlpExportMetricsServiceRequest, OtlpResourceMetrics } from './otlp-types'; +import { MetricSchemaConfig } from '../../db/types'; + +function makeDataPoint( + depName: string, + value: number, + extraAttrs: Record = {}, + timeUnixNano?: string +) { + const attributes = [ + { key: 'dependency.name', value: { stringValue: depName } }, + ...Object.entries(extraAttrs).map(([key, val]) => ({ + key, + value: { stringValue: val }, + })), + ]; + return { + attributes, + ...(timeUnixNano && { timeUnixNano }), + asDouble: value, + }; +} + +function makeRequest( + serviceName: string, + metrics: { name: string; dataPoints: ReturnType[] }[] +): OtlpExportMetricsServiceRequest { + return { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: serviceName } }], + }, + scopeMetrics: [ + { + metrics: metrics.map((m) => ({ + name: m.name, + gauge: { dataPoints: m.dataPoints }, + })), + }, + ], + }, + ], + }; +} + +describe('OtlpParser', () => { + let parser: OtlpParser; + + beforeEach(() => { + parser = new OtlpParser(); + }); + + it('parses a happy-path payload with all metrics', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [ + makeDataPoint('PostgreSQL', 0, { + 'dependency.type': 'database', + 'dependency.impact': 'critical', + 'dependency.description': 'Primary database', + }), + ], + }, + { + name: 'dependency.health.healthy', + dataPoints: [makeDataPoint('PostgreSQL', 1)], + }, + { + name: 'dependency.health.latency', + dataPoints: [makeDataPoint('PostgreSQL', 12)], + }, + { + name: 'dependency.health.code', + dataPoints: [makeDataPoint('PostgreSQL', 200)], + }, + ]); + + const results = parser.parseRequest(request); + expect(results).toHaveLength(1); + expect(results[0].serviceName).toBe('my-service'); + expect(results[0].dependencies).toHaveLength(1); + + const dep = results[0].dependencies[0]; + expect(dep.name).toBe('PostgreSQL'); + expect(dep.healthy).toBe(true); + expect(dep.health.state).toBe(0); + expect(dep.health.code).toBe(200); + expect(dep.health.latency).toBe(12); + expect(dep.type).toBe('database'); + expect(dep.impact).toBe('critical'); + expect(dep.description).toBe('Primary database'); + }); + + it('parses minimal payload with just status', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 0)], + }, + ]); + + const results = parser.parseRequest(request); + const dep = results[0].dependencies[0]; + expect(dep.name).toBe('Redis'); + expect(dep.healthy).toBe(true); + expect(dep.health.state).toBe(0); + expect(dep.health.code).toBe(200); + expect(dep.health.latency).toBe(0); + expect(dep.type).toBe('other'); + }); + + it('throws on missing service.name', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { attributes: [] }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { dataPoints: [makeDataPoint('Redis', 0)] }, + }, + ], + }, + ], + }, + ], + }; + + expect(() => parser.parseRequest(request)).toThrow( + 'OTLP payload missing required resource attribute: service.name' + ); + }); + + it('throws on missing dependency.name attribute', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { + dataPoints: [ + { + attributes: [], + asDouble: 0, + }, + ], + }, + }, + ], + }, + ], + }, + ], + }; + + expect(() => parser.parseRequest(request)).toThrow( + 'missing required attribute: dependency.name' + ); + }); + + it('groups multiple dependencies correctly', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [ + makeDataPoint('PostgreSQL', 0, { 'dependency.type': 'database' }), + makeDataPoint('Redis', 2, { 'dependency.type': 'cache' }), + ], + }, + { + name: 'dependency.health.healthy', + dataPoints: [ + makeDataPoint('PostgreSQL', 1), + makeDataPoint('Redis', 0), + ], + }, + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(2); + + const pg = results[0].dependencies.find((d) => d.name === 'PostgreSQL')!; + const redis = results[0].dependencies.find((d) => d.name === 'Redis')!; + + expect(pg.healthy).toBe(true); + expect(pg.health.state).toBe(0); + expect(pg.type).toBe('database'); + + expect(redis.healthy).toBe(false); + expect(redis.health.state).toBe(2); + expect(redis.type).toBe('cache'); + }); + + it('ignores unknown metrics', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 0)], + }, + { + name: 'some.unknown.metric', + dataPoints: [makeDataPoint('Redis', 42)], + }, + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].name).toBe('Redis'); + }); + + it('converts timeUnixNano to ISO string', () => { + // 2026-01-15T12:00:00.000Z in nanoseconds + const nanos = '1768478400000000000'; + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 0, {}, nanos)], + }, + ]); + + const results = parser.parseRequest(request); + const dep = results[0].dependencies[0]; + expect(dep.lastChecked).toBe(new Date(1768478400000).toISOString()); + }); + + it('falls back to Date.now() when no timeUnixNano', () => { + const before = Date.now(); + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 0)], + }, + ]); + + const results = parser.parseRequest(request); + const dep = results[0].dependencies[0]; + const parsed = new Date(dep.lastChecked).getTime(); + expect(parsed).toBeGreaterThanOrEqual(before); + expect(parsed).toBeLessThanOrEqual(Date.now()); + }); + + it('handles check_skipped metric', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 0)], + }, + { + name: 'dependency.health.check_skipped', + dataPoints: [makeDataPoint('Redis', 1)], + }, + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].health.skipped).toBe(true); + }); + + it('handles error_message attribute', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [ + makeDataPoint('Redis', 2, { + 'dependency.error_message': 'Connection refused', + }), + ], + }, + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].errorMessage).toBe('Connection refused'); + }); + + it('throws on invalid payload (non-object)', () => { + expect(() => parser.parseRequest(null)).toThrow('Invalid OTLP payload: expected object'); + expect(() => parser.parseRequest('bad')).toThrow('Invalid OTLP payload: expected object'); + }); + + it('throws on missing resourceMetrics array', () => { + expect(() => parser.parseRequest({})).toThrow( + 'Invalid OTLP payload: missing resourceMetrics array' + ); + }); + + it('handles asInt data point values', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.code', + gauge: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'DB' } }, + ], + asInt: '500', + }, + ], + }, + }, + ], + }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].health.code).toBe(500); + }); + + it('returns empty lastWarnings on success', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 0)], + }, + ]); + + parser.parseRequest(request); + expect(parser.lastWarnings).toEqual([]); + }); + + it('handles multiple resourceMetrics entries', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc-a' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { dataPoints: [makeDataPoint('Redis', 0)] }, + }, + ], + }, + ], + }, + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc-b' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { dataPoints: [makeDataPoint('Kafka', 2)] }, + }, + ], + }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results).toHaveLength(2); + expect(results[0].serviceName).toBe('svc-a'); + expect(results[0].dependencies[0].name).toBe('Redis'); + expect(results[1].serviceName).toBe('svc-b'); + expect(results[1].dependencies[0].name).toBe('Kafka'); + }); + + it('derives healthy=false when state is 2 and no healthy metric', () => { + const request = makeRequest('my-service', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('Redis', 2)], + }, + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].healthy).toBe(false); + }); + + it('handles empty scopeMetrics', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'svc' } }], + }, + scopeMetrics: [], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toEqual([]); + }); + + describe('public method access', () => { + it('parseResourceMetrics is callable directly', () => { + const rm: OtlpResourceMetrics = { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'direct-svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { dataPoints: [makeDataPoint('Redis', 0)] }, + }, + ], + }, + ], + }; + + const result = parser.parseResourceMetrics(rm); + expect(result.serviceName).toBe('direct-svc'); + expect(result.dependencies).toHaveLength(1); + expect(result.dependencies[0].name).toBe('Redis'); + }); + + it('extractServiceName is callable directly', () => { + const rm: OtlpResourceMetrics = { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'extracted-svc' } }], + }, + scopeMetrics: [], + }; + + expect(parser.extractServiceName(rm)).toBe('extracted-svc'); + }); + + it('extractServiceName returns undefined when missing', () => { + const rm: OtlpResourceMetrics = { + resource: { attributes: [] }, + scopeMetrics: [], + }; + + expect(parser.extractServiceName(rm)).toBeUndefined(); + }); + }); + + describe('custom MetricSchemaConfig', () => { + /** + * Helper to build data points with arbitrary attribute keys (not default dependency.name). + */ + function makeCustomDataPoint( + attrs: Record, + value: number, + timeUnixNano?: string + ) { + const attributes = Object.entries(attrs).map(([key, val]) => ({ + key, + value: { stringValue: val }, + })); + return { + attributes, + ...(timeUnixNano && { timeUnixNano }), + asDouble: value, + }; + } + + function makeCustomRequest( + serviceName: string, + metrics: { name: string; dataPoints: ReturnType[] }[] + ): OtlpExportMetricsServiceRequest { + return { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: serviceName } }], + }, + scopeMetrics: [ + { + metrics: metrics.map((m) => ({ + name: m.name, + gauge: { dataPoints: m.dataPoints }, + })), + }, + ], + }, + ], + }; + } + + it('should use custom metric names from config', () => { + const config: MetricSchemaConfig = { + metrics: { 'my.health.status': 'state' }, + labels: { 'dependency.name': 'name' }, + }; + + const request = makeCustomRequest('svc', [ + { + name: 'my.health.status', + dataPoints: [makeCustomDataPoint({ 'dependency.name': 'DB' }, 2)], + }, + ]); + + const results = parser.parseRequest(request, config); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].name).toBe('DB'); + expect(results[0].dependencies[0].health.state).toBe(2); + }); + + it('should use custom attribute names from config', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: { 'dep.name': 'name' }, + }; + + const request = makeCustomRequest('svc', [ + { + name: 'dependency.health.status', + dataPoints: [makeCustomDataPoint({ 'dep.name': 'Redis' }, 0)], + }, + ]); + + const results = parser.parseRequest(request, config); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].name).toBe('Redis'); + }); + + it('should apply latency_unit s conversion', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: {}, + latency_unit: 's', + }; + + const request = makeRequest('svc', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('DB', 0)], + }, + { + name: 'dependency.health.latency', + dataPoints: [makeDataPoint('DB', 1.5)], + }, + ]); + + const results = parser.parseRequest(request, config); + // 1.5 seconds → 1500 ms + expect(results[0].dependencies[0].health.latency).toBe(1500); + }); + + it('should default latency_unit to ms (no conversion)', () => { + const request = makeRequest('svc', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('DB', 0)], + }, + { + name: 'dependency.health.latency', + dataPoints: [makeDataPoint('DB', 42)], + }, + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].health.latency).toBe(42); + }); + + it('should merge partial overrides with defaults', () => { + // Override only the status metric name; other defaults should still work + const config: MetricSchemaConfig = { + metrics: { 'custom.status': 'state' }, + labels: {}, + }; + + const request = makeCustomRequest('svc', [ + { + name: 'custom.status', + dataPoints: [makeCustomDataPoint({ 'dependency.name': 'DB', 'dependency.type': 'database' }, 1)], + }, + { + // default latency metric still works + name: 'dependency.health.latency', + dataPoints: [makeCustomDataPoint({ 'dependency.name': 'DB' }, 55)], + }, + ]); + + const results = parser.parseRequest(request, config); + const dep = results[0].dependencies[0]; + expect(dep.health.state).toBe(1); + expect(dep.health.latency).toBe(55); + expect(dep.type).toBe('database'); + }); + + it('should pass config through parseRequest convenience method', () => { + const config: MetricSchemaConfig = { + metrics: { 'app.dep.state': 'state' }, + labels: { 'app.dep.name': 'name' }, + }; + + const request = makeCustomRequest('svc', [ + { + name: 'app.dep.state', + dataPoints: [makeCustomDataPoint({ 'app.dep.name': 'Kafka' }, 0)], + }, + ]); + + // Via parseRequest + const viaParseRequest = parser.parseRequest(request, config); + + // Via parseResourceMetrics directly + const viaResourceMetrics = parser.parseResourceMetrics( + request.resourceMetrics[0], + config + ); + + expect(viaParseRequest[0].serviceName).toBe(viaResourceMetrics.serviceName); + expect(viaParseRequest[0].dependencies).toEqual(viaResourceMetrics.dependencies); + }); + + it('should use custom attribute key in error message when name is missing', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: { 'custom.dep.name': 'name' }, + }; + + const request = makeCustomRequest('svc', [ + { + name: 'dependency.health.status', + dataPoints: [makeCustomDataPoint({}, 0)], // no name attribute + }, + ]); + + expect(() => parser.parseRequest(request, config)).toThrow( + 'missing required attribute: custom.dep.name' + ); + }); + }); + + describe('histogram data point processing', () => { + function makeHistogramRequest( + serviceName: string, + depName: string, + histogramData: { + explicitBounds: number[]; + bucketCounts: string[]; + sum?: number; + count?: string; + min?: number; + max?: number; + unit?: string; + }, + ): OtlpExportMetricsServiceRequest { + return { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: serviceName } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'http.client.request.duration', + unit: histogramData.unit, + histogram: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: depName } }, + { key: 'dependency.type', value: { stringValue: 'rest' } }, + ], + explicitBounds: histogramData.explicitBounds, + bucketCounts: histogramData.bucketCounts, + sum: histogramData.sum, + count: histogramData.count, + min: histogramData.min, + max: histogramData.max, + timeUnixNano: '1700000000000000000', + }, + ], + }, + }, + ], + }, + ], + }, + ], + }; + } + + it('produces percentiles on ProactiveDepsStatus from histogram data', () => { + const request = makeHistogramRequest('hist-svc', 'api-target', { + explicitBounds: [0.005, 0.01, 0.025, 0.05, 0.1], + bucketCounts: ['0', '0', '100', '0', '0', '0'], + count: '100', + sum: 1.75, + }); + + const results = parser.parseRequest(request); + expect(results).toHaveLength(1); + + const dep = results[0].dependencies[0]; + expect(dep.name).toBe('api-target'); + expect(dep.health.percentiles).toBeDefined(); + expect(dep.health.percentiles!.p50).toBeDefined(); + expect(dep.health.percentiles!.p95).toBeDefined(); + expect(dep.health.percentiles!.p99).toBeDefined(); + }); + + it('uses histogram avg as latency when no gauge latency is set', () => { + const request = makeHistogramRequest('hist-svc', 'api-dep', { + explicitBounds: [0.01, 0.025, 0.05], + bucketCounts: ['50', '50', '0', '0'], + count: '100', + sum: 1.5, // avg = 0.015s = 15ms + }); + + const results = parser.parseRequest(request); + const dep = results[0].dependencies[0]; + expect(dep.health.latency).toBe(15); // sum/count * 1000 + }); + + it('reads metric unit field for auto conversion', () => { + // Unit is 'ms' — no conversion needed + const request = makeHistogramRequest('unit-svc', 'dep-ms', { + explicitBounds: [10, 50, 100], + bucketCounts: ['0', '100', '0', '0'], + count: '100', + sum: 3000, // avg = 30ms + unit: 'ms', + }); + + const results = parser.parseRequest(request); + const dep = results[0].dependencies[0]; + // With unit=ms, multiplier=1, so p50 should be in [10, 50] range + expect(dep.health.percentiles!.p50).toBeGreaterThanOrEqual(10); + expect(dep.health.percentiles!.p50).toBeLessThanOrEqual(50); + }); + + it('merges histogram and gauge for same dependency correctly', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'merge-svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'postgres' } }, + { key: 'dependency.type', value: { stringValue: 'database' } }, + ], + asDouble: 0, + timeUnixNano: '1700000000000000000', + }, + ], + }, + }, + { + name: 'dependency.health.healthy', + gauge: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'postgres' } }, + ], + asDouble: 1, + }, + ], + }, + }, + { + name: 'db.client.operation.duration', + histogram: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'postgres' } }, + ], + explicitBounds: [0.01, 0.05, 0.1], + bucketCounts: ['80', '15', '5', '0'], + count: '100', + sum: 2.5, + }, + ], + }, + }, + ], + }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + const dep = results[0].dependencies[0]; + expect(dep.name).toBe('postgres'); + expect(dep.type).toBe('database'); + expect(dep.health.state).toBe(0); + expect(dep.healthy).toBe(true); + // Should have percentiles from histogram + expect(dep.health.percentiles).toBeDefined(); + expect(dep.health.percentiles!.p50).toBeDefined(); + }); + }); + + describe('sum data point processing', () => { + it('treats non-monotonic sum as gauge value', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'sum-svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.latency', + sum: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'dep-a' } }, + ], + asDouble: 42, + timeUnixNano: '1700000000000000000', + }, + ], + isMonotonic: false, + }, + }, + ], + }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + const dep = results[0].dependencies[0]; + expect(dep.name).toBe('dep-a'); + expect(dep.health.latency).toBe(42); + }); + + it('stores monotonic sum as requestCount', () => { + const request: OtlpExportMetricsServiceRequest = { + resourceMetrics: [ + { + resource: { + attributes: [{ key: 'service.name', value: { stringValue: 'counter-svc' } }], + }, + scopeMetrics: [ + { + metrics: [ + { + name: 'dependency.health.status', + gauge: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'dep-b' } }, + ], + asDouble: 0, + timeUnixNano: '1700000000000000000', + }, + ], + }, + }, + { + name: 'http.client.request.count', + sum: { + dataPoints: [ + { + attributes: [ + { key: 'dependency.name', value: { stringValue: 'dep-b' } }, + ], + asInt: '500', + timeUnixNano: '1700000000000000000', + }, + ], + isMonotonic: true, + }, + }, + ], + }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + const dep = results[0].dependencies[0]; + expect(dep.name).toBe('dep-b'); + // requestCount should be available via percentiles + expect(dep.health.percentiles).toBeDefined(); + expect(dep.health.percentiles!.requestCount).toBe(500); + }); + }); + + describe('existing gauge-only tests still pass', () => { + it('parses standard gauge payload without percentiles', () => { + const request = makeRequest('plain-svc', [ + { + name: 'dependency.health.status', + dataPoints: [makeDataPoint('db', 0)], + }, + { + name: 'dependency.health.latency', + dataPoints: [makeDataPoint('db', 25)], + }, + ]); + + const results = parser.parseRequest(request); + const dep = results[0].dependencies[0]; + expect(dep.health.latency).toBe(25); + expect(dep.health.percentiles).toBeUndefined(); + }); + }); +}); diff --git a/server/src/services/polling/OtlpParser.ts b/server/src/services/polling/OtlpParser.ts new file mode 100644 index 0000000..41ae3a6 --- /dev/null +++ b/server/src/services/polling/OtlpParser.ts @@ -0,0 +1,355 @@ +import { ProactiveDepsStatus, HealthState, DependencyType, MetricSchemaConfig } from '../../db/types'; +import { + OtlpExportMetricsServiceRequest, + OtlpResourceMetrics, + OtlpAnyValue, + OtlpNumberDataPoint, + OtlpKeyValue, +} from './otlp-types'; +import { buildEffectiveMaps, findKeyForField } from './metricSchemaUtils'; +import { computePercentiles } from '../../utils/histogramPercentiles'; + +export interface OtlpParseResult { + serviceName: string; + dependencies: ProactiveDepsStatus[]; +} + +/** Metric name → field it maps to */ +const DEFAULT_METRIC_MAP: Record = { + 'dependency.health.status': 'state', + 'dependency.health.healthy': 'healthy', + 'dependency.health.latency': 'latency', + 'dependency.health.code': 'code', + 'dependency.health.check_skipped': 'skipped', +}; + +/** Attribute key → field it maps to */ +const DEFAULT_ATTRIBUTE_MAP: Record = { + 'dependency.name': 'name', + 'dependency.type': 'type', + 'dependency.impact': 'impact', + 'dependency.description': 'description', + 'dependency.error_message': 'errorMessage', +}; + +/** + * Parses OTLP JSON metric payloads into ProactiveDepsStatus arrays. + * Extracts service.name from resource attributes and maps gauge metrics + * to dependency health fields. + */ +export class OtlpParser { + private _lastWarnings: string[] = []; + + get lastWarnings(): string[] { + return this._lastWarnings; + } + + /** + * Parse an OTLP ExportMetricsServiceRequest into per-service results. + * Each resourceMetrics entry may represent a different service. + */ + parseRequest(data: unknown, config?: MetricSchemaConfig): OtlpParseResult[] { + this._lastWarnings = []; + + if (!data || typeof data !== 'object') { + throw new Error('Invalid OTLP payload: expected object'); + } + + const request = data as OtlpExportMetricsServiceRequest; + + if (!Array.isArray(request.resourceMetrics)) { + throw new Error('Invalid OTLP payload: missing resourceMetrics array'); + } + + const results: OtlpParseResult[] = []; + + for (const rm of request.resourceMetrics) { + results.push(this.parseResourceMetrics(rm, config)); + } + + return results; + } + + parseResourceMetrics(rm: OtlpResourceMetrics, config?: MetricSchemaConfig): OtlpParseResult { + const serviceName = this.extractServiceName(rm); + + if (!serviceName) { + throw new Error('OTLP payload missing required resource attribute: service.name'); + } + + const { metricMap, labelMap, latencyUnit, healthyValue } = buildEffectiveMaps( + DEFAULT_METRIC_MAP, DEFAULT_ATTRIBUTE_MAP, config + ); + + // Collect all data points across all scope metrics, grouped by dependency name + const depMap = new Map>(); + + if (!Array.isArray(rm.scopeMetrics)) { + return { serviceName, dependencies: [] }; + } + + for (const sm of rm.scopeMetrics) { + if (!Array.isArray(sm.metrics)) continue; + + for (const metric of sm.metrics) { + // Process gauge data points (existing behavior) + const field = metricMap[metric.name]; + if (field && metric.gauge?.dataPoints) { + for (const dp of metric.gauge.dataPoints) { + const attrs = this.extractAttributes(dp, labelMap); + const depName = attrs.name as string | undefined; + + if (!depName) { + const nameAttrKey = findKeyForField(labelMap, 'name', 'dependency.name'); + throw new Error( + `OTLP data point for metric "${metric.name}" missing required attribute: ${nameAttrKey}` + ); + } + + if (!depMap.has(depName)) { + depMap.set(depName, { ...attrs }); + } + + const entry = depMap.get(depName)!; + // Merge attributes (later data points can fill in missing attrs) + for (const [k, v] of Object.entries(attrs)) { + if (entry[k] === undefined) { // eslint-disable-line security/detect-object-injection + entry[k] = v; // eslint-disable-line security/detect-object-injection + } + } + + // Set the metric value + entry[field] = this.extractDataPointValue(dp); // eslint-disable-line security/detect-object-injection + + // Capture timestamp from the data point + if (dp.timeUnixNano && !entry._timeUnixNano) { + entry._timeUnixNano = dp.timeUnixNano; + } + } + } + + // Process histogram data points — extract percentile latency + if (metric.histogram?.dataPoints) { + const unitMultiplier = this.getUnitMultiplier(metric.unit); + + for (const dp of metric.histogram.dataPoints) { + const depName = this.extractDepNameFromKeyValues(dp.attributes, labelMap); + if (!depName) continue; + + if (!depMap.has(depName)) { + depMap.set(depName, this.extractAttributesFromKeyValues(dp.attributes, labelMap)); + } + + const entry = depMap.get(depName)!; + + const bucketCounts = (dp.bucketCounts ?? []).map((c) => parseInt(c, 10) || 0); + const count = dp.count !== undefined ? parseInt(dp.count, 10) : undefined; + + const percentiles = computePercentiles( + { + explicitBounds: dp.explicitBounds ?? [], + bucketCounts, + sum: dp.sum, + count, + min: dp.min, + max: dp.max, + }, + unitMultiplier, + ); + + entry._percentiles = percentiles; + + // Use average from histogram as the latency if no gauge latency set + if (entry.latency === undefined && percentiles.avgMs > 0) { + entry.latency = Math.round(percentiles.avgMs); + } + + if (dp.timeUnixNano && !entry._timeUnixNano) { + entry._timeUnixNano = dp.timeUnixNano; + } + } + } + + // Process sum data points + if (metric.sum?.dataPoints) { + for (const dp of metric.sum.dataPoints) { + const depName = this.extractDepNameFromKeyValues(dp.attributes, labelMap); + if (!depName) continue; + + if (!depMap.has(depName)) { + depMap.set(depName, this.extractAttributesFromKeyValues(dp.attributes, labelMap)); + } + + const entry = depMap.get(depName)!; + const value = this.extractDataPointValue(dp); + + if (metric.sum!.isMonotonic === false) { + // Non-monotonic sum — treat as gauge value + const sumField = metricMap[metric.name]; + if (sumField) { + entry[sumField] = value; // eslint-disable-line security/detect-object-injection + } + } else { + // Monotonic sum — store raw count as requestCount + entry._requestCount = value; + } + + if (dp.timeUnixNano && !entry._timeUnixNano) { + entry._timeUnixNano = dp.timeUnixNano; + } + } + } + } + } + + const dependencies = Array.from(depMap.entries()).map(([name, fields]) => + this.buildDependency(name, fields, latencyUnit, healthyValue) + ); + + return { serviceName, dependencies }; + } + + extractServiceName(rm: OtlpResourceMetrics): string | undefined { + const attrs = rm.resource?.attributes; + if (!Array.isArray(attrs)) return undefined; + + for (const kv of attrs) { + if (kv.key === 'service.name') { + return this.unwrapValue(kv.value) as string | undefined; + } + } + return undefined; + } + + private extractAttributes(dp: OtlpNumberDataPoint, attrMap: Record): Record { + const result: Record = {}; + if (!Array.isArray(dp.attributes)) return result; + + for (const kv of dp.attributes) { + const field = attrMap[kv.key]; + if (field) { + result[field] = this.unwrapValue(kv.value); // eslint-disable-line security/detect-object-injection + } + } + return result; + } + + /** + * Extract attributes from OtlpKeyValue[] (used by histogram/sum data points). + */ + private extractAttributesFromKeyValues(attrs: OtlpKeyValue[] | undefined, attrMap: Record): Record { + const result: Record = {}; + if (!Array.isArray(attrs)) return result; + + for (const kv of attrs) { + const field = attrMap[kv.key]; + if (field) { + result[field] = this.unwrapValue(kv.value); // eslint-disable-line security/detect-object-injection + } + } + return result; + } + + /** + * Extract dependency name from OtlpKeyValue[] attributes. + */ + private extractDepNameFromKeyValues(attrs: OtlpKeyValue[] | undefined, attrMap: Record): string | undefined { + if (!Array.isArray(attrs)) return undefined; + + for (const kv of attrs) { + const field = attrMap[kv.key]; + if (field === 'name') { + const val = this.unwrapValue(kv.value); + return typeof val === 'string' ? val : undefined; + } + } + return undefined; + } + + /** + * Determine unit multiplier for converting metric values to milliseconds. + * OTel convention: latency in seconds → multiply by 1000 for ms. + */ + private getUnitMultiplier(unit?: string): number { + if (!unit) return 1000; // default: assume seconds (OTel convention) + const lower = unit.toLowerCase(); + if (lower === 'ms' || lower === 'milliseconds') return 1; + if (lower === 'us' || lower === 'microseconds') return 0.001; + if (lower === 'ns' || lower === 'nanoseconds') return 0.000001; + // Default to seconds → ms + return 1000; + } + + private unwrapValue(value: OtlpAnyValue | undefined): string | number | boolean | undefined { + if (!value) return undefined; + if (value.stringValue !== undefined) return value.stringValue; + if (value.intValue !== undefined) return parseInt(value.intValue, 10); + if (value.doubleValue !== undefined) return value.doubleValue; + if (value.boolValue !== undefined) return value.boolValue; + return undefined; + } + + private extractDataPointValue(dp: OtlpNumberDataPoint): number { + if (dp.asInt !== undefined) return parseInt(dp.asInt, 10); + if (dp.asDouble !== undefined) return dp.asDouble; + return 0; + } + + private buildDependency(name: string, fields: Record, latencyUnit: 'ms' | 's', healthyValue: number = 1): ProactiveDepsStatus { + const state = typeof fields.state === 'number' ? (fields.state as HealthState) : 0; + const healthy = fields.healthy !== undefined ? fields.healthy === healthyValue : state !== 2; + const rawLatency = typeof fields.latency === 'number' ? fields.latency : 0; + const latency = latencyUnit === 's' ? Math.round(rawLatency * 1000) : rawLatency; + const code = typeof fields.code === 'number' ? fields.code : 200; + const skipped = fields.skipped === 1; + + // Convert timeUnixNano to ISO string + let lastChecked: string; + if (fields._timeUnixNano && typeof fields._timeUnixNano === 'string') { + const nanos = BigInt(fields._timeUnixNano); + const millis = Number(nanos / BigInt(1_000_000)); + lastChecked = new Date(millis).toISOString(); + } else { + lastChecked = new Date().toISOString(); + } + + const depType: DependencyType = + typeof fields.type === 'string' && fields.type.trim() !== '' + ? (fields.type as DependencyType) + : 'other'; + + // Build percentiles from histogram data if present + const percentiles = fields._percentiles as { p50: number; p95: number; p99: number; min: number; max: number; count: number } | undefined; + const requestCount = typeof fields._requestCount === 'number' ? fields._requestCount : undefined; + + const healthPercentiles = percentiles || requestCount !== undefined + ? { + ...(percentiles && { + p50: percentiles.p50, + p95: percentiles.p95, + p99: percentiles.p99, + min: percentiles.min, + max: percentiles.max, + }), + ...(requestCount !== undefined && { requestCount }), + } + : undefined; + + return { + name, + description: typeof fields.description === 'string' ? fields.description : undefined, + impact: typeof fields.impact === 'string' ? fields.impact : undefined, + type: depType, + healthy, + health: { + state, + code, + latency, + ...(skipped && { skipped: true }), + ...(healthPercentiles && { percentiles: healthPercentiles }), + }, + lastChecked, + errorMessage: typeof fields.errorMessage === 'string' ? fields.errorMessage : undefined, + }; + } +} diff --git a/server/src/services/polling/PollStateManager.test.ts b/server/src/services/polling/PollStateManager.test.ts index d976826..e0e778a 100644 --- a/server/src/services/polling/PollStateManager.test.ts +++ b/server/src/services/polling/PollStateManager.test.ts @@ -22,6 +22,7 @@ describe('PollStateManager', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: new Date().toISOString(), updated_at: new Date().toISOString(), ...overrides, diff --git a/server/src/services/polling/PrometheusParser.test.ts b/server/src/services/polling/PrometheusParser.test.ts new file mode 100644 index 0000000..2537cb9 --- /dev/null +++ b/server/src/services/polling/PrometheusParser.test.ts @@ -0,0 +1,351 @@ +import { MetricSchemaConfig } from '../../db/types'; +import { PrometheusParser } from './PrometheusParser'; + +describe('PrometheusParser', () => { + let parser: PrometheusParser; + + beforeEach(() => { + parser = new PrometheusParser(); + }); + + it('parses a happy-path payload with all metrics', () => { + const text = [ + '# HELP dependency_health_status Health status of dependencies', + '# TYPE dependency_health_status gauge', + 'dependency_health_status{name="PostgreSQL",type="database",impact="critical",description="Primary database"} 0', + 'dependency_health_healthy{name="PostgreSQL"} 1', + 'dependency_health_latency_ms{name="PostgreSQL"} 12', + 'dependency_health_code{name="PostgreSQL"} 200', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + + const dep = deps[0]; + expect(dep.name).toBe('PostgreSQL'); + expect(dep.healthy).toBe(true); + expect(dep.health.state).toBe(0); + expect(dep.health.code).toBe(200); + expect(dep.health.latency).toBe(12); + expect(dep.type).toBe('database'); + expect(dep.impact).toBe('critical'); + expect(dep.description).toBe('Primary database'); + }); + + it('parses minimal payload (status only)', () => { + const text = 'dependency_health_status{name="Redis"} 0\n'; + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis'); + expect(deps[0].healthy).toBe(true); + expect(deps[0].health.state).toBe(0); + expect(deps[0].health.code).toBe(200); + expect(deps[0].health.latency).toBe(0); + expect(deps[0].type).toBe('other'); + }); + + it('warns and skips metrics with missing name label', () => { + const text = [ + 'dependency_health_status{type="database"} 0', + 'dependency_health_status{name="Redis"} 0', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis'); + expect(parser.lastWarnings).toHaveLength(1); + expect(parser.lastWarnings[0]).toContain('missing required "name" label'); + }); + + it('parses multiple dependencies', () => { + const text = [ + 'dependency_health_status{name="PostgreSQL",type="database"} 0', + 'dependency_health_status{name="Redis",type="cache"} 2', + 'dependency_health_healthy{name="PostgreSQL"} 1', + 'dependency_health_healthy{name="Redis"} 0', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toHaveLength(2); + + const pg = deps.find((d) => d.name === 'PostgreSQL')!; + const redis = deps.find((d) => d.name === 'Redis')!; + + expect(pg.healthy).toBe(true); + expect(pg.type).toBe('database'); + expect(redis.healthy).toBe(false); + expect(redis.type).toBe('cache'); + }); + + it('uses raw latency value when unit is ms (default)', () => { + const text = [ + 'dependency_health_status{name="Redis"} 0', + 'dependency_health_latency_ms{name="Redis"} 45', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps[0].health.latency).toBe(45); + }); + + it('skips # HELP and # TYPE lines', () => { + const text = [ + '# HELP dependency_health_status Help text', + '# TYPE dependency_health_status gauge', + 'dependency_health_status{name="Redis"} 0', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis'); + }); + + it('skips unknown metrics silently', () => { + const text = [ + 'process_cpu_seconds_total 42.5', + 'dependency_health_status{name="Redis"} 0', + 'go_goroutines 15', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis'); + expect(parser.lastWarnings).toHaveLength(0); + }); + + it('handles malformed lines gracefully', () => { + const text = [ + 'this is not a valid metric line !!', + 'dependency_health_status{name="Redis"} 0', + 'another bad line', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis'); + expect(parser.lastWarnings).toHaveLength(2); + expect(parser.lastWarnings[0]).toContain('malformed metric line'); + }); + + it('handles check_skipped metric', () => { + const text = [ + 'dependency_health_status{name="Redis"} 0', + 'dependency_health_check_skipped{name="Redis"} 1', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps[0].health.skipped).toBe(true); + }); + + it('handles error_message label', () => { + const text = + 'dependency_health_status{name="Redis",error_message="Connection refused"} 2\n'; + + const deps = parser.parse(text); + expect(deps[0].errorMessage).toBe('Connection refused'); + }); + + it('throws on non-string input', () => { + expect(() => parser.parse(42 as unknown as string)).toThrow( + 'Invalid Prometheus payload: expected string' + ); + }); + + it('returns empty array for empty input', () => { + const deps = parser.parse(''); + expect(deps).toEqual([]); + }); + + it('returns empty array for comments-only input', () => { + const text = [ + '# HELP some_metric Help text', + '# TYPE some_metric gauge', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps).toEqual([]); + }); + + it('handles metrics without labels (no braces)', () => { + // Unknown metric without labels — should be skipped silently + const text = 'process_cpu_seconds_total 42.5\n'; + + const deps = parser.parse(text); + expect(deps).toEqual([]); + }); + + it('handles metric lines with timestamps', () => { + const text = 'dependency_health_status{name="Redis"} 0 1768478400000\n'; + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis'); + }); + + it('derives healthy=false when state is 2 and no healthy metric', () => { + const text = 'dependency_health_status{name="Redis"} 2\n'; + + const deps = parser.parse(text); + expect(deps[0].healthy).toBe(false); + expect(deps[0].health.state).toBe(2); + }); + + it('derives healthy=true when state is 1 (warning)', () => { + const text = 'dependency_health_status{name="Redis"} 1\n'; + + const deps = parser.parse(text); + expect(deps[0].healthy).toBe(true); + expect(deps[0].health.state).toBe(1); + }); + + it('clears warnings between parse calls', () => { + parser.parse('bad line here'); + expect(parser.lastWarnings.length).toBeGreaterThan(0); + + parser.parse('dependency_health_status{name="Redis"} 0'); + expect(parser.lastWarnings).toHaveLength(0); + }); + + it('handles labels with escaped quotes', () => { + const text = 'dependency_health_status{name="Redis \\"Primary\\""} 0\n'; + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('Redis "Primary"'); + }); + + it('handles labels with commas in values', () => { + const text = + 'dependency_health_status{name="Redis",description="Cache, primary"} 0\n'; + + const deps = parser.parse(text); + expect(deps).toHaveLength(1); + expect(deps[0].description).toBe('Cache, primary'); + }); + + describe('custom MetricSchemaConfig', () => { + it('should use custom metric names from config', () => { + const config: MetricSchemaConfig = { + metrics: { my_dep_status: 'state' }, + labels: {}, + }; + + const text = 'my_dep_status{name="x"} 2\n'; + const deps = parser.parse(text, config); + + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('x'); + expect(deps[0].health.state).toBe(2); + }); + + it('should use custom label names from config', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: { dependency: 'name' }, + }; + + const text = 'dependency_health_status{dependency="x"} 0\n'; + const deps = parser.parse(text, config); + + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('x'); + }); + + it('should apply latency_unit s conversion', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: {}, + latency_unit: 's', + }; + + const text = [ + 'dependency_health_status{name="Redis"} 0', + 'dependency_health_latency_ms{name="Redis"} 0.045', + ].join('\n'); + + const deps = parser.parse(text, config); + expect(deps[0].health.latency).toBe(45); // 0.045s * 1000 + }); + + it('should default latency_unit to ms (no conversion)', () => { + const text = [ + 'dependency_health_status{name="Redis"} 0', + 'dependency_health_latency_ms{name="Redis"} 45', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps[0].health.latency).toBe(45); + }); + + it('should merge partial overrides with defaults', () => { + const config: MetricSchemaConfig = { + metrics: { my_custom_state: 'state' }, + labels: {}, + }; + + const text = [ + 'my_custom_state{name="Redis"} 1', + 'dependency_health_healthy{name="Redis"} 1', + 'dependency_health_latency_ms{name="Redis"} 30', + ].join('\n'); + + const deps = parser.parse(text, config); + expect(deps).toHaveLength(1); + expect(deps[0].health.state).toBe(1); + expect(deps[0].healthy).toBe(true); + expect(deps[0].health.latency).toBe(30); + }); + + it('should use defaults when config has empty metrics/labels', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: {}, + }; + + const text = [ + 'dependency_health_status{name="PostgreSQL",type="database"} 0', + 'dependency_health_healthy{name="PostgreSQL"} 1', + 'dependency_health_latency_ms{name="PostgreSQL"} 12', + ].join('\n'); + + const deps = parser.parse(text, config); + expect(deps).toHaveLength(1); + expect(deps[0].name).toBe('PostgreSQL'); + expect(deps[0].health.state).toBe(0); + expect(deps[0].healthy).toBe(true); + expect(deps[0].health.latency).toBe(12); + expect(deps[0].type).toBe('database'); + }); + + it('should use healthy_value to determine healthy status', () => { + const config: MetricSchemaConfig = { + metrics: { dependency_health: 'healthy' }, + labels: { dependency: 'name' }, + healthy_value: 0, + }; + + const text = [ + 'dependency_health{dependency="auth-svc"} 0', + 'dependency_health{dependency="broken-svc"} 2', + ].join('\n'); + + const deps = parser.parse(text, config); + expect(deps).toHaveLength(2); + + const auth = deps.find(d => d.name === 'auth-svc')!; + expect(auth.healthy).toBe(true); + + const broken = deps.find(d => d.name === 'broken-svc')!; + expect(broken.healthy).toBe(false); + }); + + it('should default healthy_value to 1 when not specified', () => { + const text = [ + 'dependency_health_healthy{name="svc"} 1', + ].join('\n'); + + const deps = parser.parse(text); + expect(deps[0].healthy).toBe(true); + }); + }); +}); diff --git a/server/src/services/polling/PrometheusParser.ts b/server/src/services/polling/PrometheusParser.ts new file mode 100644 index 0000000..3c00e30 --- /dev/null +++ b/server/src/services/polling/PrometheusParser.ts @@ -0,0 +1,255 @@ +import { ProactiveDepsStatus, HealthState, DependencyType, MetricSchemaConfig } from '../../db/types'; +import { buildEffectiveMaps, findKeyForField } from './metricSchemaUtils'; + +/** Metric name → field it maps to */ +const DEFAULT_METRIC_MAP: Record = { + dependency_health_status: 'state', + dependency_health_healthy: 'healthy', + dependency_health_latency_ms: 'latency', + dependency_health_code: 'code', + dependency_health_check_skipped: 'skipped', +}; + +/** Label name → field it maps to */ +const DEFAULT_LABEL_MAP: Record = { + name: 'name', + type: 'type', + impact: 'impact', + description: 'description', + error_message: 'errorMessage', +}; + +interface ParsedLine { + metricName: string; + labels: Record; + value: number; +} + +/** + * Parses Prometheus text exposition format into ProactiveDepsStatus arrays. + * Extracts dependency health metrics from Prometheus-style metric lines. + */ +export class PrometheusParser { + private _lastWarnings: string[] = []; + private latencyUnit: 'ms' | 's' = 'ms'; + private healthyValue: number = 1; + + get lastWarnings(): string[] { + return this._lastWarnings; + } + + /** + * Parse Prometheus text exposition format. + * @param text - Raw Prometheus metrics text + * @returns Array of parsed dependency statuses + */ + parse(text: string, config?: MetricSchemaConfig): ProactiveDepsStatus[] { + this._lastWarnings = []; + + const { metricMap, labelMap, latencyUnit, healthyValue } = buildEffectiveMaps( + DEFAULT_METRIC_MAP, DEFAULT_LABEL_MAP, config + ); + this.latencyUnit = latencyUnit; + this.healthyValue = healthyValue; + + if (typeof text !== 'string') { + throw new Error('Invalid Prometheus payload: expected string'); + } + + const lines = text.split('\n'); + const depMap = new Map>(); + + for (const line of lines) { + const trimmed = line.trim(); + + // Skip empty lines and comments (# HELP, # TYPE, etc.) + if (trimmed === '' || trimmed.startsWith('#')) { + continue; + } + + const parsed = this.parseLine(trimmed); + if (!parsed) { + this._lastWarnings.push(`Skipping malformed metric line: ${trimmed}`); + continue; + } + + const field = metricMap[parsed.metricName]; + if (!field) { + // Unknown metric — skip silently + continue; + } + + // Extract dependency name from labels + const nameKey = findKeyForField(labelMap, 'name', 'name'); + const depName = parsed.labels[nameKey]; // eslint-disable-line security/detect-object-injection + if (!depName) { + this._lastWarnings.push( + `Metric "${parsed.metricName}" missing required "${nameKey}" label, skipping` + ); + continue; + } + + // Initialize entry if first time seeing this dependency + if (!depMap.has(depName)) { + const attrs: Record = { name: depName }; + // Extract optional labels + for (const [labelKey, labelValue] of Object.entries(parsed.labels)) { + const mappedField = labelMap[labelKey]; // eslint-disable-line security/detect-object-injection + if (mappedField && mappedField !== 'name') { + attrs[mappedField] = labelValue; // eslint-disable-line security/detect-object-injection + } + } + depMap.set(depName, attrs); + } else { + // Merge any new labels from this line + const entry = depMap.get(depName)!; + for (const [labelKey, labelValue] of Object.entries(parsed.labels)) { + const mappedField = labelMap[labelKey]; // eslint-disable-line security/detect-object-injection + if (mappedField && mappedField !== 'name' && entry[mappedField] === undefined) { // eslint-disable-line security/detect-object-injection + entry[mappedField] = labelValue; // eslint-disable-line security/detect-object-injection + } + } + } + + const entry = depMap.get(depName)!; + entry[field] = parsed.value; // eslint-disable-line security/detect-object-injection + } + + return Array.from(depMap.entries()).map(([name, fields]) => + this.buildDependency(name, fields) + ); + } + + /** + * Parse a single Prometheus metric line. + * Format: metric_name{label1="value1",label2="value2"} value [timestamp] + * or: metric_name value [timestamp] + */ + private parseLine(line: string): ParsedLine | null { + // Match: metricName{labels} value OR metricName value + const braceIndex = line.indexOf('{'); + + let metricName: string; + let labelsStr: string; + let rest: string; + + if (braceIndex !== -1) { + metricName = line.substring(0, braceIndex).trim(); + const closeBrace = line.indexOf('}', braceIndex); + if (closeBrace === -1) return null; + labelsStr = line.substring(braceIndex + 1, closeBrace); + rest = line.substring(closeBrace + 1).trim(); + } else { + // No labels + const spaceIndex = line.indexOf(' '); + if (spaceIndex === -1) return null; + metricName = line.substring(0, spaceIndex).trim(); + labelsStr = ''; + rest = line.substring(spaceIndex + 1).trim(); + } + + if (!metricName) return null; + + // Parse the value (first token of rest, ignore optional timestamp) + const valueStr = rest.split(/\s+/)[0]; + const value = parseFloat(valueStr); + if (isNaN(value)) return null; + + // Parse labels + const labels = this.parseLabels(labelsStr); + + return { metricName, labels, value }; + } + + /** + * Parse label string: key1="value1",key2="value2" + */ + private parseLabels(labelsStr: string): Record { + const labels: Record = {}; + if (!labelsStr.trim()) return labels; + + // State machine to handle commas inside quoted values + let key = ''; + let value = ''; + let inValue = false; + let escaped = false; + + for (let i = 0; i < labelsStr.length; i++) { + const ch = labelsStr[i]; // eslint-disable-line security/detect-object-injection + + if (escaped) { + value += ch; + escaped = false; + continue; + } + + if (ch === '\\' && inValue) { + escaped = true; + continue; + } + + if (ch === '"') { + inValue = !inValue; + continue; + } + + if (ch === '=' && !inValue) { + // Transition from key to value + continue; + } + + if (ch === ',' && !inValue) { + // End of label pair + if (key.trim()) { + labels[key.trim()] = value; + } + key = ''; + value = ''; + continue; + } + + if (inValue) { + value += ch; + } else { + key += ch; + } + } + + // Last pair + if (key.trim()) { + labels[key.trim()] = value; + } + + return labels; + } + + private buildDependency(name: string, fields: Record): ProactiveDepsStatus { + const state = typeof fields.state === 'number' ? (fields.state as HealthState) : 0; + const healthy = fields.healthy !== undefined ? fields.healthy === this.healthyValue : state !== 2; + const rawLatency = typeof fields.latency === 'number' ? fields.latency : 0; + const latency = this.latencyUnit === 's' ? Math.round(rawLatency * 1000) : rawLatency; + const code = typeof fields.code === 'number' ? fields.code : 200; + const skipped = fields.skipped === 1; + + const depType: DependencyType = + typeof fields.type === 'string' && fields.type.trim() !== '' + ? (fields.type as DependencyType) + : 'other'; + + return { + name, + description: typeof fields.description === 'string' ? fields.description : undefined, + impact: typeof fields.impact === 'string' ? fields.impact : undefined, + type: depType, + healthy, + health: { + state, + code, + latency, + ...(skipped && { skipped: true }), + }, + lastChecked: new Date().toISOString(), + errorMessage: typeof fields.errorMessage === 'string' ? fields.errorMessage : undefined, + }; + } +} diff --git a/server/src/services/polling/ServicePoller.test.ts b/server/src/services/polling/ServicePoller.test.ts index 9fa112f..d2d5537 100644 --- a/server/src/services/polling/ServicePoller.test.ts +++ b/server/src/services/polling/ServicePoller.test.ts @@ -34,6 +34,7 @@ describe('ServicePoller', () => { manifest_managed: 0, manifest_config_id: null, manifest_last_synced_values: null, + health_endpoint_format: 'default', created_at: new Date().toISOString(), updated_at: new Date().toISOString(), ...overrides, @@ -216,4 +217,105 @@ describe('ServicePoller', () => { expect(poller.serviceName).toBe('Updated Service'); }); }); + + describe('format-aware fetching', () => { + it('should use Accept: text/plain for prometheus format', async () => { + const service = createService({ health_endpoint_format: 'prometheus' }); + const poller = new ServicePoller(service, mockParser, mockUpsertService); + + mockFetch.mockResolvedValueOnce({ + ok: true, + text: jest.fn().mockResolvedValue('dependency_health_status{name="db"} 0'), + }); + + await poller.poll(); + + expect(mockFetch).toHaveBeenCalledWith( + 'http://test-service/health', + expect.objectContaining({ + headers: expect.objectContaining({ + 'Accept': 'text/plain; version=0.0.4', + }), + }) + ); + }); + + it('should parse prometheus response as text not JSON', async () => { + const promText = 'dependency_health_status{name="db"} 0'; + const mockTextFn = jest.fn().mockResolvedValue(promText); + const service = createService({ health_endpoint_format: 'prometheus' }); + const poller = new ServicePoller(service, mockParser, mockUpsertService); + + mockFetch.mockResolvedValueOnce({ + ok: true, + text: mockTextFn, + json: jest.fn(), + }); + + await poller.poll(); + + expect(mockTextFn).toHaveBeenCalled(); + }); + + it('should use Accept: application/json for default format', async () => { + const service = createService({ health_endpoint_format: 'default' }); + const poller = new ServicePoller(service, mockParser, mockUpsertService); + + mockFetch.mockResolvedValueOnce({ + ok: true, + json: jest.fn().mockResolvedValue([]), + }); + + await poller.poll(); + + expect(mockFetch).toHaveBeenCalledWith( + 'http://test-service/health', + expect.objectContaining({ + headers: expect.objectContaining({ + 'Accept': 'application/json', + }), + }) + ); + }); + + it('should use Accept: application/json for schema format', async () => { + const service = createService({ health_endpoint_format: 'schema' }); + const poller = new ServicePoller(service, mockParser, mockUpsertService); + + mockFetch.mockResolvedValueOnce({ + ok: true, + json: jest.fn().mockResolvedValue({}), + }); + + await poller.poll(); + + expect(mockFetch).toHaveBeenCalledWith( + 'http://test-service/health', + expect.objectContaining({ + headers: expect.objectContaining({ + 'Accept': 'application/json', + }), + }) + ); + }); + + it('should pass format to parser.parse()', async () => { + const service = createService({ health_endpoint_format: 'prometheus' }); + const poller = new ServicePoller(service, mockParser, mockUpsertService); + + mockFetch.mockResolvedValueOnce({ + ok: true, + text: jest.fn().mockResolvedValue(''), + }); + + await poller.poll(); + + expect(mockParser.parse).toHaveBeenCalledWith( + '', + null, + 'Test Service', + 'prometheus' + ); + }); + }); }); diff --git a/server/src/services/polling/ServicePoller.ts b/server/src/services/polling/ServicePoller.ts index 31f52fc..0ada7d9 100644 --- a/server/src/services/polling/ServicePoller.ts +++ b/server/src/services/polling/ServicePoller.ts @@ -1,4 +1,4 @@ -import { Service, ProactiveDepsStatus, SchemaMapping } from '../../db/types'; +import { Service, ProactiveDepsStatus, SchemaMapping, MetricSchemaConfig } from '../../db/types'; import { ExponentialBackoff } from './backoff'; import { PollResult } from './types'; import { DependencyParser, getDependencyParser } from './DependencyParser'; @@ -74,15 +74,15 @@ export class ServicePoller { } /** - * Parse the service's schema_config JSON string into a SchemaMapping object. + * Parse the service's schema_config JSON string into a SchemaMapping or MetricSchemaConfig. * Returns null if no schema config is set or if parsing fails. */ - private getSchemaConfig(): SchemaMapping | null { + private getSchemaConfig(): SchemaMapping | MetricSchemaConfig | null { if (!this.service.schema_config) { return null; } try { - return JSON.parse(this.service.schema_config) as SchemaMapping; + return JSON.parse(this.service.schema_config); } catch { return null; } @@ -92,6 +92,9 @@ export class ServicePoller { // Validate URL against private/internal IPs (DNS rebinding protection) await validateUrlNotPrivate(this.service.health_endpoint); + const format = this.service.health_endpoint_format ?? 'default'; + const isPrometheus = format === 'prometheus'; + const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), POLL_TIMEOUT_MS); @@ -99,7 +102,7 @@ export class ServicePoller { const response = await fetch(this.service.health_endpoint, { method: 'GET', headers: { - 'Accept': 'application/json', + 'Accept': isPrometheus ? 'text/plain; version=0.0.4' : 'application/json', 'User-Agent': 'Dependencies-Dashboard/1.0', }, signal: controller.signal, @@ -109,9 +112,9 @@ export class ServicePoller { throw new Error(`HTTP ${response.status}: ${response.statusText}`); } - const data = await response.json(); + const data = isPrometheus ? await response.text() : await response.json(); const schemaConfig = this.getSchemaConfig(); - return this.parser.parse(data, schemaConfig, this.service.name); + return this.parser.parse(data, schemaConfig, this.service.name, format); } finally { clearTimeout(timeout); } diff --git a/server/src/services/polling/TraceDependencyBridge.test.ts b/server/src/services/polling/TraceDependencyBridge.test.ts new file mode 100644 index 0000000..3a94c14 --- /dev/null +++ b/server/src/services/polling/TraceDependencyBridge.test.ts @@ -0,0 +1,145 @@ +import { TraceDependencyBridge } from './TraceDependencyBridge'; +import { TraceDependency } from './TraceParser'; + +/** Helper to build a TraceDependency with defaults */ +function makeDep(overrides: Partial = {}): TraceDependency { + return { + targetName: 'postgres', + type: 'database', + latencyMs: 50, + isError: false, + spanKind: 3, // CLIENT + description: 'SELECT users', + attributes: {}, + ...overrides, + }; +} + +describe('TraceDependencyBridge', () => { + let bridge: TraceDependencyBridge; + + beforeEach(() => { + bridge = new TraceDependencyBridge(); + }); + + describe('bridgeToDepsStatus', () => { + it('maps isError=false to state=0 (OK) and healthy=true', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ isError: false })]); + + expect(result).toHaveLength(1); + expect(result[0].health.state).toBe(0); + expect(result[0].healthy).toBe(true); + }); + + it('maps isError=true to state=2 (CRITICAL) and healthy=false', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ isError: true })]); + + expect(result).toHaveLength(1); + expect(result[0].health.state).toBe(2); + expect(result[0].healthy).toBe(false); + }); + + it('sets health.code to 200 for non-error and 500 for error', () => { + const ok = bridge.bridgeToDepsStatus([makeDep({ isError: false })]); + const err = bridge.bridgeToDepsStatus([makeDep({ isError: true })]); + + expect(ok[0].health.code).toBe(200); + expect(err[0].health.code).toBe(500); + }); + + it('sets discovery_source to otlp_trace', () => { + const result = bridge.bridgeToDepsStatus([makeDep()]); + + expect(result[0].discovery_source).toBe('otlp_trace'); + }); + + it('maps span duration to health.latency', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ latencyMs: 123 })]); + + expect(result[0].health.latency).toBe(123); + }); + + it('maps targetName to name', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ targetName: 'redis' })]); + + expect(result[0].name).toBe('redis'); + }); + + it('maps type through to output', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ type: 'cache' })]); + + expect(result[0].type).toBe('cache'); + }); + + it('maps description through to output', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ description: 'GET key' })]); + + expect(result[0].description).toBe('GET key'); + }); + + it('sets lastChecked to current time', () => { + const before = new Date().toISOString(); + const result = bridge.bridgeToDepsStatus([makeDep()]); + const after = new Date().toISOString(); + + expect(result[0].lastChecked >= before).toBe(true); + expect(result[0].lastChecked <= after).toBe(true); + }); + + it('returns empty array for empty input', () => { + const result = bridge.bridgeToDepsStatus([]); + expect(result).toHaveLength(0); + }); + + it('converts multiple distinct dependencies', () => { + const result = bridge.bridgeToDepsStatus([ + makeDep({ targetName: 'postgres', type: 'database', latencyMs: 10 }), + makeDep({ targetName: 'redis', type: 'cache', latencyMs: 5 }), + makeDep({ targetName: 'kafka', type: 'message_queue', latencyMs: 20 }), + ]); + + expect(result).toHaveLength(3); + expect(result.map((r) => r.name).sort()).toEqual(['kafka', 'postgres', 'redis']); + }); + + it('groups multiple spans to same target with avg latency', () => { + const result = bridge.bridgeToDepsStatus([ + makeDep({ targetName: 'postgres', latencyMs: 40 }), + makeDep({ targetName: 'postgres', latencyMs: 60 }), + ]); + + expect(result).toHaveLength(1); + expect(result[0].name).toBe('postgres'); + expect(result[0].health.latency).toBe(50); // avg of 40 and 60 + }); + + it('groups multiple spans to same target with any-error-wins', () => { + const result = bridge.bridgeToDepsStatus([ + makeDep({ targetName: 'postgres', isError: false }), + makeDep({ targetName: 'postgres', isError: true }), + ]); + + expect(result).toHaveLength(1); + expect(result[0].healthy).toBe(false); + expect(result[0].health.state).toBe(2); + expect(result[0].health.code).toBe(500); + }); + + it('keeps healthy=true when all grouped spans are non-error', () => { + const result = bridge.bridgeToDepsStatus([ + makeDep({ targetName: 'redis', isError: false }), + makeDep({ targetName: 'redis', isError: false }), + ]); + + expect(result).toHaveLength(1); + expect(result[0].healthy).toBe(true); + expect(result[0].health.state).toBe(0); + }); + + it('sets description to undefined for empty string', () => { + const result = bridge.bridgeToDepsStatus([makeDep({ description: '' })]); + + expect(result[0].description).toBeUndefined(); + }); + }); +}); diff --git a/server/src/services/polling/TraceDependencyBridge.ts b/server/src/services/polling/TraceDependencyBridge.ts new file mode 100644 index 0000000..4cc1e11 --- /dev/null +++ b/server/src/services/polling/TraceDependencyBridge.ts @@ -0,0 +1,75 @@ +import { ProactiveDepsStatus, HealthState } from '../../db/types'; +import { TraceDependency } from './TraceParser'; + +/** + * Extended ProactiveDepsStatus that carries the discovery source for + * trace-discovered dependencies. The trace route reads this field + * when building DependencyUpsertInput. + */ +export interface TraceBridgedDepsStatus extends ProactiveDepsStatus { + discovery_source: 'otlp_trace'; +} + +/** + * Thin adapter converting TraceDependency[] (from TraceParser) to + * ProactiveDepsStatus[] for the existing DependencyUpsertService pipeline. + * + * Deduplicates by target name: averages latency, any-error-wins. + */ +export class TraceDependencyBridge { + /** + * Convert trace-extracted dependencies into the ProactiveDepsStatus shape + * consumed by DependencyUpsertService.upsert(). + */ + bridgeToDepsStatus(traceDeps: TraceDependency[]): TraceBridgedDepsStatus[] { + // Group by targetName for deduplication (handles merged results) + const grouped = new Map< + string, + { + totalLatency: number; + count: number; + isError: boolean; + type: string; + description: string; + } + >(); + + for (const dep of traceDeps) { + const existing = grouped.get(dep.targetName); + if (existing) { + existing.totalLatency += dep.latencyMs; + existing.count += 1; + if (dep.isError) existing.isError = true; + } else { + grouped.set(dep.targetName, { + totalLatency: dep.latencyMs, + count: 1, + isError: dep.isError, + type: dep.type, + description: dep.description, + }); + } + } + + const now = new Date().toISOString(); + + return Array.from(grouped.entries()).map(([name, agg]) => { + const state: HealthState = agg.isError ? 2 : 0; + const latency = Math.round(agg.totalLatency / agg.count); + + return { + name, + description: agg.description || undefined, + type: agg.type, + healthy: !agg.isError, + health: { + state, + code: agg.isError ? 500 : 200, + latency, + }, + lastChecked: now, + discovery_source: 'otlp_trace' as const, + }; + }); + } +} diff --git a/server/src/services/polling/TraceParser.test.ts b/server/src/services/polling/TraceParser.test.ts new file mode 100644 index 0000000..a816d1c --- /dev/null +++ b/server/src/services/polling/TraceParser.test.ts @@ -0,0 +1,729 @@ +import { TraceParser, SpanKind, SpanStatusCode } from './TraceParser'; +import { OtlpExportTraceServiceRequest, OtlpResourceSpans, OtlpSpan, OtlpKeyValue } from './otlp-types'; + +/** Helper to build an OtlpKeyValue from a plain key/value */ +function kv(key: string, value: string | number | boolean): OtlpKeyValue { + if (typeof value === 'string') return { key, value: { stringValue: value } }; + if (typeof value === 'number') { + return Number.isInteger(value) + ? { key, value: { intValue: String(value) } } + : { key, value: { doubleValue: value } }; + } + return { key, value: { boolValue: value } }; +} + +/** Helper to build a span with defaults */ +function makeSpan(overrides: Partial & { attributes?: OtlpKeyValue[] } = {}): OtlpSpan { + return { + traceId: 'abc123', + spanId: 'span1', + name: 'test-span', + kind: SpanKind.CLIENT, + startTimeUnixNano: '1700000000000000000', // some timestamp + endTimeUnixNano: '1700000000050000000', // +50ms + ...overrides, + }; +} + +/** Helper to build a full trace request */ +function makeTraceRequest( + serviceName: string, + spans: OtlpSpan[] +): OtlpExportTraceServiceRequest { + return { + resourceSpans: [ + { + resource: { + attributes: [kv('service.name', serviceName)], + }, + scopeSpans: [{ spans }], + }, + ], + }; +} + +describe('TraceParser', () => { + let parser: TraceParser; + + beforeEach(() => { + parser = new TraceParser(); + }); + + describe('parseRequest', () => { + it('throws on non-object payload', () => { + expect(() => parser.parseRequest(null)).toThrow('Invalid OTLP trace payload: expected object'); + expect(() => parser.parseRequest('bad')).toThrow('Invalid OTLP trace payload: expected object'); + }); + + it('throws on missing resourceSpans array', () => { + expect(() => parser.parseRequest({})).toThrow( + 'Invalid OTLP trace payload: missing resourceSpans array' + ); + }); + + it('throws on missing service.name', () => { + const request: OtlpExportTraceServiceRequest = { + resourceSpans: [ + { + resource: { attributes: [] }, + scopeSpans: [{ spans: [makeSpan()] }], + }, + ], + }; + expect(() => parser.parseRequest(request)).toThrow( + 'OTLP trace payload missing required resource attribute: service.name' + ); + }); + + it('parses multiple resourceSpans entries', () => { + const request: OtlpExportTraceServiceRequest = { + resourceSpans: [ + { + resource: { attributes: [kv('service.name', 'svc-a')] }, + scopeSpans: [ + { + spans: [ + makeSpan({ attributes: [kv('peer.service', 'db-a')] }), + ], + }, + ], + }, + { + resource: { attributes: [kv('service.name', 'svc-b')] }, + scopeSpans: [ + { + spans: [ + makeSpan({ spanId: 'span2', attributes: [kv('peer.service', 'db-b')] }), + ], + }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results).toHaveLength(2); + expect(results[0].serviceName).toBe('svc-a'); + expect(results[0].dependencies[0].targetName).toBe('db-a'); + expect(results[1].serviceName).toBe('svc-b'); + expect(results[1].dependencies[0].targetName).toBe('db-b'); + }); + }); + + describe('span kind filtering', () => { + it('extracts dependencies from CLIENT spans', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ kind: SpanKind.CLIENT, attributes: [kv('peer.service', 'postgres')] }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].targetName).toBe('postgres'); + }); + + it('extracts dependencies from PRODUCER spans', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + kind: SpanKind.PRODUCER, + attributes: [kv('messaging.system', 'kafka')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].targetName).toBe('kafka'); + }); + + it('ignores SERVER spans for dependency discovery', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ kind: SpanKind.SERVER, attributes: [kv('peer.service', 'caller')] }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(0); + }); + + it('ignores INTERNAL spans for dependency discovery', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ kind: SpanKind.INTERNAL, attributes: [kv('peer.service', 'internal')] }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(0); + }); + + it('ignores CONSUMER spans for dependency discovery', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ kind: SpanKind.CONSUMER, attributes: [kv('messaging.system', 'kafka')] }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(0); + }); + + it('ignores UNSPECIFIED kind spans', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ kind: SpanKind.UNSPECIFIED, attributes: [kv('peer.service', 'something')] }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(0); + }); + }); + + describe('target name resolution', () => { + it('resolves via peer.service first', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('peer.service', 'my-database'), + kv('db.system', 'postgresql'), + kv('server.address', 'db.example.com'), + ], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('my-database'); + }); + + it('falls back to db.system', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('db.system', 'postgresql'), + kv('server.address', 'db.example.com'), + ], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('postgresql'); + }); + + it('falls back to db.system.name', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('db.system.name', 'mysql')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('mysql'); + }); + + it('falls back to messaging.system', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + kind: SpanKind.PRODUCER, + attributes: [kv('messaging.system', 'rabbitmq')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('rabbitmq'); + }); + + it('falls back to rpc.system', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('rpc.system', 'grpc')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('grpc'); + }); + + it('falls back to rpc.system.name', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('rpc.system.name', 'grpc')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('grpc'); + }); + + it('falls back to server.address', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('server.address', 'api.example.com')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('api.example.com'); + }); + + it('falls back to hostname from url.full', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('url.full', 'https://api.example.com/v1/users')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies[0].targetName).toBe('api.example.com'); + }); + + it('warns and skips spans with no resolvable target name', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ spanId: 'orphan', attributes: [] }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(0); + expect(parser.lastWarnings).toHaveLength(1); + expect(parser.lastWarnings[0]).toContain('orphan'); + expect(parser.lastWarnings[0]).toContain('no resolvable target name'); + }); + }); + + describe('dependency type inference', () => { + it('infers database from db.system', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ attributes: [kv('db.system', 'postgresql'), kv('peer.service', 'pg')] }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('database'); + }); + + it('infers cache for redis', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ attributes: [kv('db.system', 'redis'), kv('peer.service', 'cache')] }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('cache'); + }); + + it('infers cache for memcached', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ attributes: [kv('db.system', 'memcached'), kv('peer.service', 'mc')] }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('cache'); + }); + + it('infers message_queue from messaging.system', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + kind: SpanKind.PRODUCER, + attributes: [kv('messaging.system', 'kafka')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('message_queue'); + }); + + it('infers grpc from rpc.system=grpc', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('rpc.system', 'grpc'), kv('peer.service', 'user-svc')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('grpc'); + }); + + it('infers rest from http.request.method', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('http.request.method', 'GET'), + kv('server.address', 'api.example.com'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('rest'); + }); + + it('infers rest from legacy http.method', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('http.method', 'POST'), + kv('server.address', 'api.example.com'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('rest'); + }); + + it('defaults to other when no known system attribute', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('peer.service', 'unknown-thing')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.type).toBe('other'); + }); + }); + + describe('description generation', () => { + it('generates HTTP description with method and host/path', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('http.request.method', 'GET'), + kv('server.address', 'api.example.com'), + kv('url.path', '/v1/users'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.description).toBe('GET api.example.com/v1/users'); + }); + + it('generates DB description with operation and namespace.collection', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('db.system', 'postgresql'), + kv('db.operation', 'SELECT'), + kv('db.namespace', 'public'), + kv('db.collection.name', 'users'), + kv('peer.service', 'pg'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.description).toBe('SELECT public.users'); + }); + + it('generates messaging description with operation and destination', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + kind: SpanKind.PRODUCER, + attributes: [ + kv('messaging.system', 'kafka'), + kv('messaging.operation', 'publish'), + kv('messaging.destination.name', 'orders-topic'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.description).toBe('publish orders-topic'); + }); + + it('generates gRPC description with service/method', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('rpc.system', 'grpc'), + kv('rpc.service', 'UserService'), + kv('rpc.method', 'GetUser'), + kv('peer.service', 'user-svc'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.description).toBe('UserService/GetUser'); + }); + + it('falls back to span name when no attributes match', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + name: 'custom-operation', + attributes: [kv('peer.service', 'something')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.description).toBe('custom-operation'); + }); + + it('generates DB description falling back to db.system name', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('db.system', 'postgresql'), + kv('peer.service', 'pg'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.description).toBe('postgresql'); + }); + }); + + describe('latency computation', () => { + it('computes latency from nanosecond timestamps', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000000150000000', // +150ms + attributes: [kv('peer.service', 'target')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.latencyMs).toBe(150); + }); + + it('handles sub-millisecond durations', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000000000500000', // +0.5ms → rounds to 0 + attributes: [kv('peer.service', 'target')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.latencyMs).toBe(0); + }); + + it('returns 0 for invalid timestamps', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + startTimeUnixNano: 'not-a-number', + endTimeUnixNano: 'also-not', + attributes: [kv('peer.service', 'target')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.latencyMs).toBe(0); + }); + }); + + describe('deduplication by target name', () => { + it('deduplicates by target name with average latency', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + spanId: 'span1', + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000000100000000', // 100ms + attributes: [kv('peer.service', 'postgres')], + }), + makeSpan({ + spanId: 'span2', + startTimeUnixNano: '1700000000000000000', + endTimeUnixNano: '1700000000200000000', // 200ms + attributes: [kv('peer.service', 'postgres')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].targetName).toBe('postgres'); + expect(results[0].dependencies[0].latencyMs).toBe(150); // avg of 100 and 200 + }); + + it('deduplicates with any-error-wins', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + spanId: 'span1', + attributes: [kv('peer.service', 'postgres')], + status: { code: SpanStatusCode.OK }, + }), + makeSpan({ + spanId: 'span2', + attributes: [kv('peer.service', 'postgres')], + status: { code: SpanStatusCode.ERROR, message: 'connection refused' }, + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(1); + expect(results[0].dependencies[0].isError).toBe(true); + }); + + it('keeps distinct targets as separate dependencies', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + spanId: 'span1', + attributes: [kv('peer.service', 'postgres')], + }), + makeSpan({ + spanId: 'span2', + attributes: [kv('peer.service', 'redis')], + }), + ]); + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(2); + const names = results[0].dependencies.map((d) => d.targetName).sort(); + expect(names).toEqual(['postgres', 'redis']); + }); + }); + + describe('error detection', () => { + it('maps status code ERROR to isError=true', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('peer.service', 'target')], + status: { code: SpanStatusCode.ERROR }, + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.isError).toBe(true); + }); + + it('maps status code OK to isError=false', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('peer.service', 'target')], + status: { code: SpanStatusCode.OK }, + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.isError).toBe(false); + }); + + it('maps status code UNSET to isError=false', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('peer.service', 'target')], + status: { code: SpanStatusCode.UNSET }, + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.isError).toBe(false); + }); + + it('maps missing status to isError=false', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('peer.service', 'target')], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.isError).toBe(false); + }); + }); + + describe('edge cases', () => { + it('handles empty scopeSpans', () => { + const request: OtlpExportTraceServiceRequest = { + resourceSpans: [ + { + resource: { attributes: [kv('service.name', 'svc')] }, + scopeSpans: [], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toEqual([]); + }); + + it('handles empty spans array', () => { + const request: OtlpExportTraceServiceRequest = { + resourceSpans: [ + { + resource: { attributes: [kv('service.name', 'svc')] }, + scopeSpans: [{ spans: [] }], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toEqual([]); + }); + + it('handles multiple scopeSpans within one resource', () => { + const request: OtlpExportTraceServiceRequest = { + resourceSpans: [ + { + resource: { attributes: [kv('service.name', 'svc')] }, + scopeSpans: [ + { spans: [makeSpan({ spanId: 's1', attributes: [kv('peer.service', 'db-a')] })] }, + { spans: [makeSpan({ spanId: 's2', attributes: [kv('peer.service', 'db-b')] })] }, + ], + }, + ], + }; + + const results = parser.parseRequest(request); + expect(results[0].dependencies).toHaveLength(2); + }); + + it('resets warnings between calls', () => { + // First call with unresolvable span + const req1 = makeTraceRequest('svc', [makeSpan({ attributes: [] })]); + parser.parseRequest(req1); + expect(parser.lastWarnings.length).toBeGreaterThan(0); + + // Second call — warnings reset + const req2 = makeTraceRequest('svc', [ + makeSpan({ attributes: [kv('peer.service', 'ok')] }), + ]); + parser.parseRequest(req2); + expect(parser.lastWarnings).toHaveLength(0); + }); + + it('extractServiceName returns undefined when no resource attributes', () => { + const rs: OtlpResourceSpans = { + resource: {}, + scopeSpans: [], + }; + expect(parser.extractServiceName(rs)).toBeUndefined(); + }); + + it('parseResourceSpans is callable directly', () => { + const rs: OtlpResourceSpans = { + resource: { attributes: [kv('service.name', 'direct-svc')] }, + scopeSpans: [ + { spans: [makeSpan({ attributes: [kv('peer.service', 'target')] })] }, + ], + }; + + const result = parser.parseResourceSpans(rs); + expect(result.serviceName).toBe('direct-svc'); + expect(result.dependencies).toHaveLength(1); + }); + + it('captures span attributes on the dependency', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [ + kv('peer.service', 'postgres'), + kv('db.system', 'postgresql'), + kv('db.operation', 'SELECT'), + ], + }), + ]); + + const dep = parser.parseRequest(request)[0].dependencies[0]; + expect(dep.attributes['peer.service']).toBe('postgres'); + expect(dep.attributes['db.system']).toBe('postgresql'); + expect(dep.attributes['db.operation']).toBe('SELECT'); + }); + + it('handles invalid url.full gracefully', () => { + const request = makeTraceRequest('my-svc', [ + makeSpan({ + attributes: [kv('url.full', 'not-a-valid-url')], + }), + ]); + + const results = parser.parseRequest(request); + // Can't resolve target — generates warning + expect(results[0].dependencies).toHaveLength(0); + expect(parser.lastWarnings).toHaveLength(1); + }); + }); +}); diff --git a/server/src/services/polling/TraceParser.ts b/server/src/services/polling/TraceParser.ts new file mode 100644 index 0000000..85b8ac8 --- /dev/null +++ b/server/src/services/polling/TraceParser.ts @@ -0,0 +1,361 @@ +import { DependencyType } from '../../db/types'; +import { + OtlpExportTraceServiceRequest, + OtlpResourceSpans, + OtlpSpan, + OtlpAnyValue, + OtlpKeyValue, +} from './otlp-types'; + +/** Span kind constants per OTel spec */ +export const SpanKind = { + UNSPECIFIED: 0, + INTERNAL: 1, + SERVER: 2, + CLIENT: 3, + PRODUCER: 4, + CONSUMER: 5, +} as const; + +/** OTel status code constants */ +export const SpanStatusCode = { + UNSET: 0, + OK: 1, + ERROR: 2, +} as const; + +/** Cache systems that map to 'cache' type instead of 'database' */ +const CACHE_SYSTEMS = new Set(['redis', 'memcached', 'valkey']); + +export interface TraceDependency { + targetName: string; + type: DependencyType; + latencyMs: number; + isError: boolean; + spanKind: number; + description: string; + attributes: Record; +} + +export interface TraceDependencyResult { + serviceName: string; + dependencies: TraceDependency[]; +} + +/** + * Parses OTLP trace payloads, extracting dependency information from + * CLIENT and PRODUCER spans. Mirrors OtlpParser's structure for metrics. + */ +export class TraceParser { + private _lastWarnings: string[] = []; + + get lastWarnings(): string[] { + return this._lastWarnings; + } + + /** + * Parse an OTLP ExportTraceServiceRequest into per-service results. + * Each resourceSpans entry may represent a different service. + */ + parseRequest(data: unknown): TraceDependencyResult[] { + this._lastWarnings = []; + + if (!data || typeof data !== 'object') { + throw new Error('Invalid OTLP trace payload: expected object'); + } + + const request = data as OtlpExportTraceServiceRequest; + + if (!Array.isArray(request.resourceSpans)) { + throw new Error('Invalid OTLP trace payload: missing resourceSpans array'); + } + + const results: TraceDependencyResult[] = []; + + for (const rs of request.resourceSpans) { + results.push(this.parseResourceSpans(rs)); + } + + return results; + } + + /** + * Parse a single resourceSpans entry into a TraceDependencyResult. + * Only CLIENT (kind=3) and PRODUCER (kind=4) spans produce dependencies. + */ + parseResourceSpans(rs: OtlpResourceSpans): TraceDependencyResult { + const serviceName = this.extractServiceName(rs); + + if (!serviceName) { + throw new Error('OTLP trace payload missing required resource attribute: service.name'); + } + + const allSpans = this.collectSpans(rs); + + // Filter to CLIENT and PRODUCER spans for dependency discovery + const depSpans = allSpans.filter( + (s) => s.kind === SpanKind.CLIENT || s.kind === SpanKind.PRODUCER + ); + + // Build dependencies, deduplicating by target name + const depMap = new Map }>(); + + for (const span of depSpans) { + const targetName = this.resolveTargetName(span); + if (!targetName) { + this._lastWarnings.push( + `Span "${span.name}" (${span.spanId}) has no resolvable target name — skipped for dependency discovery` + ); + continue; + } + + const latencyMs = this.computeLatencyMs(span); + const isError = span.status?.code === SpanStatusCode.ERROR; + const type = this.inferDependencyType(span); + const description = this.generateDescription(span); + const attributes = this.extractSpanAttributes(span); + + const existing = depMap.get(targetName); + if (existing) { + // Aggregate: average latency, any-error-wins + existing.totalLatency += latencyMs; + existing.count += 1; + if (isError) existing.isError = true; + } else { + depMap.set(targetName, { + totalLatency: latencyMs, + count: 1, + isError, + type, + spanKind: span.kind ?? SpanKind.UNSPECIFIED, + description, + attributes, + }); + } + } + + const dependencies: TraceDependency[] = Array.from(depMap.entries()).map( + ([targetName, agg]) => ({ + targetName, + type: agg.type, + latencyMs: Math.round(agg.totalLatency / agg.count), + isError: agg.isError, + spanKind: agg.spanKind, + description: agg.description, + attributes: agg.attributes, + }) + ); + + return { serviceName, dependencies }; + } + + /** + * Extract service.name from resource attributes. + * Reuses the same pattern as OtlpParser. + */ + extractServiceName(rs: OtlpResourceSpans): string | undefined { + const attrs = rs.resource?.attributes; + if (!Array.isArray(attrs)) return undefined; + + for (const kv of attrs) { + if (kv.key === 'service.name') { + return this.unwrapValue(kv.value) as string | undefined; + } + } + return undefined; + } + + /** + * Collect all spans from all scope spans in a resource. + */ + private collectSpans(rs: OtlpResourceSpans): OtlpSpan[] { + const spans: OtlpSpan[] = []; + if (!Array.isArray(rs.scopeSpans)) return spans; + + for (const ss of rs.scopeSpans) { + if (!Array.isArray(ss.spans)) continue; + spans.push(...ss.spans); + } + return spans; + } + + /** + * Resolve the target dependency name from span attributes. + * Priority chain: peer.service → db.system/db.system.name → + * messaging.system → rpc.system/rpc.system.name → server.address → + * hostname from url.full + */ + private resolveTargetName(span: OtlpSpan): string | undefined { + const attrs = this.buildAttrMap(span.attributes); + + // 1. peer.service — explicit dependency name + const peerService = attrs['peer.service']; + if (peerService) return String(peerService); + + // 2. db.system / db.system.name — database system + const dbSystem = attrs['db.system'] ?? attrs['db.system.name']; + if (dbSystem) return String(dbSystem); + + // 3. messaging.system — message broker + const messagingSystem = attrs['messaging.system']; + if (messagingSystem) return String(messagingSystem); + + // 4. rpc.system / rpc.system.name — RPC framework + const rpcSystem = attrs['rpc.system'] ?? attrs['rpc.system.name']; + if (rpcSystem) return String(rpcSystem); + + // 5. server.address — target host + const serverAddress = attrs['server.address']; + if (serverAddress) return String(serverAddress); + + // 6. hostname from url.full + const urlFull = attrs['url.full']; + if (urlFull) { + try { + const url = new URL(String(urlFull)); + return url.hostname; + } catch { + // Invalid URL — skip + } + } + + return undefined; + } + + /** + * Infer the dependency type from span attributes. + * db.system → database/cache, messaging → message_queue, + * rpc grpc → grpc, http → rest, else other + */ + private inferDependencyType(span: OtlpSpan): DependencyType { + const attrs = this.buildAttrMap(span.attributes); + + // Database or cache + const dbSystem = attrs['db.system'] ?? attrs['db.system.name']; + if (dbSystem) { + return CACHE_SYSTEMS.has(String(dbSystem).toLowerCase()) ? 'cache' : 'database'; + } + + // Message queue + if (attrs['messaging.system']) return 'message_queue'; + + // gRPC + const rpcSystem = attrs['rpc.system'] ?? attrs['rpc.system.name']; + if (rpcSystem && String(rpcSystem).toLowerCase() === 'grpc') return 'grpc'; + + // HTTP/REST + if (attrs['http.request.method'] || attrs['http.method']) return 'rest'; + + return 'other'; + } + + /** + * Auto-generate a human-readable description from span attributes. + * HTTP: "{method} {host}{path}", DB: "{op} {namespace}.{collection}", + * messaging: "{op} {destination}", gRPC: "{rpc.method}" + */ + private generateDescription(span: OtlpSpan): string { + const attrs = this.buildAttrMap(span.attributes); + + // HTTP description + const httpMethod = attrs['http.request.method'] ?? attrs['http.method']; + if (httpMethod) { + const host = attrs['server.address'] ?? attrs['net.peer.name'] ?? ''; + const path = attrs['url.path'] ?? attrs['http.target'] ?? ''; + const parts = [String(httpMethod)]; + if (host || path) parts.push(`${host}${path}`); + return parts.join(' '); + } + + // DB description + const dbSystem = attrs['db.system'] ?? attrs['db.system.name']; + if (dbSystem) { + const op = attrs['db.operation'] ?? attrs['db.operation.name'] ?? ''; + const ns = attrs['db.namespace'] ?? attrs['db.name'] ?? ''; + const collection = attrs['db.collection.name'] ?? attrs['db.sql.table'] ?? ''; + const parts: string[] = []; + if (op) parts.push(String(op)); + const target = [ns, collection].filter(Boolean).join('.'); + if (target) parts.push(target); + return parts.join(' ') || String(dbSystem); + } + + // Messaging description + const messagingSystem = attrs['messaging.system']; + if (messagingSystem) { + const op = attrs['messaging.operation'] ?? attrs['messaging.operation.name'] ?? ''; + const dest = attrs['messaging.destination.name'] ?? attrs['messaging.destination'] ?? ''; + const parts: string[] = []; + if (op) parts.push(String(op)); + if (dest) parts.push(String(dest)); + return parts.join(' ') || String(messagingSystem); + } + + // gRPC description + const rpcMethod = attrs['rpc.method']; + if (rpcMethod) { + const rpcService = attrs['rpc.service'] ?? ''; + return rpcService ? `${rpcService}/${rpcMethod}` : String(rpcMethod); + } + + // Fallback to span name + return span.name; + } + + /** + * Compute span duration in milliseconds from nanosecond timestamps. + * Uses BigInt for precision with large nanosecond values. + */ + private computeLatencyMs(span: OtlpSpan): number { + try { + const startNanos = BigInt(span.startTimeUnixNano); + const endNanos = BigInt(span.endTimeUnixNano); + const durationNanos = endNanos - startNanos; + return Number(durationNanos / BigInt(1_000_000)); + } catch { + return 0; + } + } + + /** + * Build a flat key→value map from OtlpKeyValue attributes. + */ + private buildAttrMap( + attributes?: OtlpKeyValue[] + ): Record { + const map: Record = {}; + if (!Array.isArray(attributes)) return map; + + for (const kv of attributes) { + const val = this.unwrapValue(kv.value); + if (val !== undefined) { + map[kv.key] = val; + } + } + return map; + } + + /** + * Extract interesting span attributes as a flat record. + * Captures the attributes that were used for resolution. + */ + private extractSpanAttributes( + span: OtlpSpan + ): Record { + return this.buildAttrMap(span.attributes); + } + + /** + * Unwrap an OtlpAnyValue to a primitive. Same logic as OtlpParser. + */ + private unwrapValue( + value: OtlpAnyValue | undefined + ): string | number | boolean | undefined { + if (!value) return undefined; + if (value.stringValue !== undefined) return value.stringValue; + if (value.intValue !== undefined) return parseInt(value.intValue, 10); + if (value.doubleValue !== undefined) return value.doubleValue; + if (value.boolValue !== undefined) return value.boolValue; + return undefined; + } +} diff --git a/server/src/services/polling/metricSchemaUtils.test.ts b/server/src/services/polling/metricSchemaUtils.test.ts new file mode 100644 index 0000000..3c24cd7 --- /dev/null +++ b/server/src/services/polling/metricSchemaUtils.test.ts @@ -0,0 +1,125 @@ +import { buildEffectiveMaps, findKeyForField, isMetricSchemaConfig } from './metricSchemaUtils'; +import { MetricSchemaConfig, SchemaMapping } from '../../db/types'; + +const DEFAULT_METRICS: Record = { + 'dep_status': 'state', + 'dep_healthy': 'healthy', + 'dep_latency': 'latency', +}; + +const DEFAULT_LABELS: Record = { + 'name': 'name', + 'type': 'type', +}; + +describe('buildEffectiveMaps', () => { + it('should return defaults when no config is provided', () => { + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS); + expect(result.metricMap).toEqual(DEFAULT_METRICS); + expect(result.labelMap).toEqual(DEFAULT_LABELS); + expect(result.latencyUnit).toBe('ms'); + }); + + it('should return defaults when config has empty metrics and labels', () => { + const config: MetricSchemaConfig = { metrics: {}, labels: {} }; + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS, config); + expect(result.metricMap).toEqual(DEFAULT_METRICS); + expect(result.labelMap).toEqual(DEFAULT_LABELS); + }); + + it('should override a metric mapping', () => { + const config: MetricSchemaConfig = { + metrics: { my_custom_status: 'state' }, + labels: {}, + }; + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS, config); + expect(result.metricMap.my_custom_status).toBe('state'); + expect(result.metricMap.dep_status).toBeUndefined(); + // Other defaults preserved + expect(result.metricMap.dep_healthy).toBe('healthy'); + expect(result.metricMap.dep_latency).toBe('latency'); + }); + + it('should override a label mapping', () => { + const config: MetricSchemaConfig = { + metrics: {}, + labels: { dependency: 'name' }, + }; + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS, config); + expect(result.labelMap.dependency).toBe('name'); + expect(result.labelMap.name).toBeUndefined(); + // Other default preserved + expect(result.labelMap.type).toBe('type'); + }); + + it('should handle multiple overrides', () => { + const config: MetricSchemaConfig = { + metrics: { my_status: 'state', my_latency: 'latency' }, + labels: { dep: 'name', dep_type: 'type' }, + }; + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS, config); + expect(result.metricMap).toEqual({ + dep_healthy: 'healthy', + my_status: 'state', + my_latency: 'latency', + }); + expect(result.labelMap).toEqual({ + dep: 'name', + dep_type: 'type', + }); + }); + + it('should respect latency_unit from config', () => { + const config: MetricSchemaConfig = { metrics: {}, labels: {}, latency_unit: 's' }; + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS, config); + expect(result.latencyUnit).toBe('s'); + }); + + it('should default latency_unit to ms', () => { + const config: MetricSchemaConfig = { metrics: {}, labels: {} }; + const result = buildEffectiveMaps(DEFAULT_METRICS, DEFAULT_LABELS, config); + expect(result.latencyUnit).toBe('ms'); + }); +}); + +describe('findKeyForField', () => { + it('should find the key that maps to a given field', () => { + expect(findKeyForField(DEFAULT_LABELS, 'name', 'fallback')).toBe('name'); + expect(findKeyForField(DEFAULT_LABELS, 'type', 'fallback')).toBe('type'); + }); + + it('should return fallback when field not found', () => { + expect(findKeyForField(DEFAULT_LABELS, 'missing', 'my_fallback')).toBe('my_fallback'); + }); + + it('should find custom keys from overridden maps', () => { + const customMap = { dependency: 'name', dep_type: 'type' }; + expect(findKeyForField(customMap, 'name', 'fallback')).toBe('dependency'); + }); +}); + +describe('isMetricSchemaConfig', () => { + it('should return true for objects with metrics key', () => { + expect(isMetricSchemaConfig({ metrics: {}, labels: {} })).toBe(true); + }); + + it('should return true for objects with labels key only', () => { + expect(isMetricSchemaConfig({ labels: { dep: 'name' } } as unknown as MetricSchemaConfig)).toBe(true); + }); + + it('should return false for SchemaMapping', () => { + const schema: SchemaMapping = { + root: 'data', + fields: { name: 'n', healthy: 'h' }, + }; + expect(isMetricSchemaConfig(schema)).toBe(false); + }); + + it('should return false for null', () => { + expect(isMetricSchemaConfig(null)).toBe(false); + }); + + it('should return false for undefined', () => { + expect(isMetricSchemaConfig(undefined)).toBe(false); + }); +}); diff --git a/server/src/services/polling/metricSchemaUtils.ts b/server/src/services/polling/metricSchemaUtils.ts new file mode 100644 index 0000000..06c53ec --- /dev/null +++ b/server/src/services/polling/metricSchemaUtils.ts @@ -0,0 +1,83 @@ +import { MetricSchemaConfig, SchemaMapping } from '../../db/types'; + +export interface EffectiveMaps { + metricMap: Record; + labelMap: Record; + latencyUnit: 'ms' | 's'; + healthyValue: number; +} + +/** + * Build effective metric and label maps by merging user overrides into defaults. + * When a user overrides a target field, the default entry for that field is removed + * and replaced with the user's mapping. + */ +export function buildEffectiveMaps( + defaultMetrics: Record, + defaultLabels: Record, + config?: MetricSchemaConfig, +): EffectiveMaps { + if (!config) { + return { + metricMap: { ...defaultMetrics }, + labelMap: { ...defaultLabels }, + latencyUnit: 'ms', + healthyValue: 1, + }; + } + + const metricMap = { ...defaultMetrics }; + const labelMap = { ...defaultLabels }; + + // Override metric mappings + if (config.metrics && Object.keys(config.metrics).length > 0) { + const overriddenFields = new Set(Object.values(config.metrics)); + for (const [key, field] of Object.entries(metricMap)) { + if (overriddenFields.has(field)) { + delete metricMap[key]; // eslint-disable-line security/detect-object-injection + } + } + Object.assign(metricMap, config.metrics); + } + + // Override label mappings + if (config.labels && Object.keys(config.labels).length > 0) { + const overriddenFields = new Set(Object.values(config.labels)); + for (const [key, field] of Object.entries(labelMap)) { + if (overriddenFields.has(field)) { + delete labelMap[key]; // eslint-disable-line security/detect-object-injection + } + } + Object.assign(labelMap, config.labels); + } + + return { + metricMap, + labelMap, + latencyUnit: config.latency_unit ?? 'ms', + healthyValue: config.healthy_value ?? 1, + }; +} + +/** + * Find the key in a map that maps to a given target field. + */ +export function findKeyForField( + map: Record, + field: string, + fallback: string, +): string { + const entry = Object.entries(map).find(([, f]) => f === field); + return entry ? entry[0] : fallback; +} + +/** + * Type guard to distinguish MetricSchemaConfig from SchemaMapping. + * MetricSchemaConfig has metrics+labels, SchemaMapping has root+fields. + */ +export function isMetricSchemaConfig( + config: SchemaMapping | MetricSchemaConfig | null | undefined +): config is MetricSchemaConfig { + if (!config || typeof config !== 'object') return false; + return 'metrics' in config || 'labels' in config; +} diff --git a/server/src/services/polling/otlp-types.ts b/server/src/services/polling/otlp-types.ts new file mode 100644 index 0000000..758eb74 --- /dev/null +++ b/server/src/services/polling/otlp-types.ts @@ -0,0 +1,125 @@ +/** + * TypeScript type definitions for the OTLP JSON export structure. + * Based on the OpenTelemetry Protocol (OTLP) specification for metrics. + * @see https://opentelemetry.io/docs/specs/otlp/#otlphttp + */ + +export interface OtlpAnyValue { + stringValue?: string; + intValue?: string; // OTLP encodes int64 as string in JSON + doubleValue?: number; + boolValue?: boolean; + arrayValue?: { values: OtlpAnyValue[] }; + kvlistValue?: { values: OtlpKeyValue[] }; +} + +export interface OtlpKeyValue { + key: string; + value: OtlpAnyValue; +} + +export interface OtlpNumberDataPoint { + attributes?: OtlpKeyValue[]; + startTimeUnixNano?: string; + timeUnixNano?: string; + asInt?: string; // int64 encoded as string + asDouble?: number; +} + +export interface OtlpGauge { + dataPoints: OtlpNumberDataPoint[]; +} + +// Histogram types (DPS-110h) +export interface OtlpHistogramDataPoint { + attributes?: OtlpKeyValue[]; + startTimeUnixNano?: string; + timeUnixNano?: string; + count?: string; // uint64 encoded as string + sum?: number; + min?: number; + max?: number; + bucketCounts: string[]; // uint64[] encoded as strings + explicitBounds: number[]; +} + +export interface OtlpHistogram { + dataPoints: OtlpHistogramDataPoint[]; + aggregationTemporality?: number; // 1=DELTA, 2=CUMULATIVE +} + +// Sum types (DPS-110h) +export interface OtlpSum { + dataPoints: OtlpNumberDataPoint[]; + aggregationTemporality?: number; // 1=DELTA, 2=CUMULATIVE + isMonotonic?: boolean; +} + +export interface OtlpMetric { + name: string; + description?: string; + unit?: string; + gauge?: OtlpGauge; + histogram?: OtlpHistogram; + sum?: OtlpSum; +} + +export interface OtlpScopeMetrics { + scope?: { + name?: string; + version?: string; + attributes?: OtlpKeyValue[]; + }; + metrics: OtlpMetric[]; +} + +export interface OtlpResourceMetrics { + resource?: { + attributes?: OtlpKeyValue[]; + }; + scopeMetrics: OtlpScopeMetrics[]; +} + +export interface OtlpExportMetricsServiceRequest { + resourceMetrics: OtlpResourceMetrics[]; +} + +// OTLP Trace types (DPS-110g) + +export interface OtlpSpanStatus { + code?: number; // 0=UNSET, 1=OK, 2=ERROR + message?: string; +} + +/** Span kind enum: 0=UNSPECIFIED, 1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER */ +export interface OtlpSpan { + traceId: string; + spanId: string; + parentSpanId?: string; + name: string; + kind?: number; + startTimeUnixNano: string; + endTimeUnixNano: string; + attributes?: OtlpKeyValue[]; + status?: OtlpSpanStatus; +} + +export interface OtlpScopeSpans { + scope?: { + name?: string; + version?: string; + attributes?: OtlpKeyValue[]; + }; + spans: OtlpSpan[]; +} + +export interface OtlpResourceSpans { + resource?: { + attributes?: OtlpKeyValue[]; + }; + scopeSpans: OtlpScopeSpans[]; +} + +export interface OtlpExportTraceServiceRequest { + resourceSpans: OtlpResourceSpans[]; +} diff --git a/server/src/services/polling/otlpServiceResolver.ts b/server/src/services/polling/otlpServiceResolver.ts new file mode 100644 index 0000000..66b893c --- /dev/null +++ b/server/src/services/polling/otlpServiceResolver.ts @@ -0,0 +1,39 @@ +import { StoreRegistry } from '../../stores'; +import { Service } from '../../db/types'; +import logger from '../../utils/logger'; + +/** + * Find a service by name + team, or auto-create it as an OTLP push service. + * Shared between the metrics and traces OTLP receiver routes. + */ +export function findOrCreateService( + stores: StoreRegistry, + teamId: string, + serviceName: string, + warnings: string[], +): Service { + const teamServices = stores.services.findByTeamId(teamId); + const existing = teamServices.find((s) => s.name === serviceName); + + if (existing) { + if (existing.health_endpoint_format !== 'otlp') { + warnings.push( + `Service "${serviceName}" exists with format "${existing.health_endpoint_format}" — receiving OTLP data but not overwriting format` + ); + } + return existing; + } + + // Auto-register new service + const service = stores.services.create({ + name: serviceName, + team_id: teamId, + health_endpoint: '', + health_endpoint_format: 'otlp', + poll_interval_ms: 0, + }); + + logger.info({ serviceId: service.id, serviceName, teamId }, 'auto-registered OTLP service'); + + return service; +} diff --git a/server/src/services/retention/DataRetentionService.test.ts b/server/src/services/retention/DataRetentionService.test.ts index 2e0526f..a6dc7be 100644 --- a/server/src/services/retention/DataRetentionService.test.ts +++ b/server/src/services/retention/DataRetentionService.test.ts @@ -10,6 +10,12 @@ const mockDeletePollHistory = jest.fn().mockReturnValue(0); const mockDeleteSyncHistory = jest.fn().mockReturnValue(0); const mockDeleteDriftFlags = jest.fn().mockReturnValue(0); const mockDeleteExpiredMutes = jest.fn().mockReturnValue(0); +const mockPruneMinuteBuckets = jest.fn().mockReturnValue(0); +const mockPruneHourBuckets = jest.fn().mockReturnValue(0); +const mockPruneOrphanedBuckets = jest.fn().mockReturnValue(0); +const mockDeleteSpans = jest.fn().mockReturnValue(0); +const mockDeleteOldDismissed = jest.fn().mockReturnValue(0); +const mockAppSettingsGet = jest.fn(); const mockSettingsStore = {}; jest.mock('../../stores', () => ({ @@ -23,6 +29,14 @@ jest.mock('../../stores', () => ({ manifestSyncHistory: { deleteOlderThan: mockDeleteSyncHistory }, driftFlags: { deleteOlderThan: mockDeleteDriftFlags }, alertMutes: { deleteExpired: mockDeleteExpiredMutes }, + apiKeyUsage: { + pruneMinuteBuckets: mockPruneMinuteBuckets, + pruneHourBuckets: mockPruneHourBuckets, + pruneOrphanedBuckets: mockPruneOrphanedBuckets, + }, + spans: { deleteOlderThan: mockDeleteSpans }, + associations: { deleteOldDismissed: mockDeleteOldDismissed }, + appSettings: { get: mockAppSettingsGet }, settings: mockSettingsStore, }), })); @@ -63,6 +77,9 @@ describe('DataRetentionService', () => { mockDeletePollHistory.mockReturnValue(0); mockDeleteSyncHistory.mockReturnValue(0); mockDeleteDriftFlags.mockReturnValue(0); + mockDeleteSpans.mockReturnValue(0); + mockDeleteOldDismissed.mockReturnValue(0); + mockAppSettingsGet.mockReturnValue('7'); // Default settings mockGet.mockImplementation((key) => { @@ -376,6 +393,126 @@ describe('DataRetentionService', () => { }); }); + describe('span retention cleanup', () => { + it('should delete spans older than configured retention days', () => { + mockAppSettingsGet.mockReturnValue('3'); + mockDeleteSpans.mockReturnValue(42); + + const now = new Date('2026-02-21T10:00:00Z'); + jest.setSystemTime(now); + + const service = DataRetentionService.getInstance(); + const result = service.runCleanup(); + + expect(result.spansDeleted).toBe(42); + expect(mockDeleteSpans).toHaveBeenCalledWith(expect.stringMatching(/^\d{4}-\d{2}-\d{2}T/)); + + // Verify the cutoff is approximately 3 days ago + const callArg = mockDeleteSpans.mock.calls[0][0]; + const cutoff = new Date(callArg); + const expected3DaysAgo = new Date(now); + expected3DaysAgo.setDate(expected3DaysAgo.getDate() - 3); + expect(Math.abs(cutoff.getTime() - expected3DaysAgo.getTime())).toBeLessThan(1000); + }); + + it('should default to 7 days when app_settings has no value', () => { + mockAppSettingsGet.mockReturnValue(undefined); + + const now = new Date('2026-02-21T10:00:00Z'); + jest.setSystemTime(now); + + const service = DataRetentionService.getInstance(); + service.runCleanup(); + + const callArg = mockDeleteSpans.mock.calls[0][0]; + const cutoff = new Date(callArg); + const expected7DaysAgo = new Date(now); + expected7DaysAgo.setDate(expected7DaysAgo.getDate() - 7); + expect(Math.abs(cutoff.getTime() - expected7DaysAgo.getTime())).toBeLessThan(1000); + }); + + it('should use span retention window independent of data_retention_days', () => { + mockGet.mockImplementation((key) => { + if (key === 'data_retention_days') return 30; + if (key === 'retention_cleanup_time') return '02:00'; + return undefined; + }); + mockAppSettingsGet.mockReturnValue('1'); + + const now = new Date('2026-02-21T10:00:00Z'); + jest.setSystemTime(now); + + const service = DataRetentionService.getInstance(); + service.runCleanup(); + + // Regular tables use 30-day cutoff + const regularCutoff = new Date(mockDeleteLatency.mock.calls[0][0]); + const expected30DaysAgo = new Date(now); + expected30DaysAgo.setDate(expected30DaysAgo.getDate() - 30); + expect(Math.abs(regularCutoff.getTime() - expected30DaysAgo.getTime())).toBeLessThan(1000); + + // Spans use 1-day cutoff + const spanCutoff = new Date(mockDeleteSpans.mock.calls[0][0]); + const expected1DayAgo = new Date(now); + expected1DayAgo.setDate(expected1DayAgo.getDate() - 1); + expect(Math.abs(spanCutoff.getTime() - expected1DayAgo.getTime())).toBeLessThan(1000); + }); + + it('should include spansDeleted in log output', () => { + mockDeleteSpans.mockReturnValue(100); + + const service = DataRetentionService.getInstance(); + service.runCleanup(); + + expect(logger.info).toHaveBeenCalledWith( + expect.objectContaining({ spansDeleted: 100 }), + 'data retention cleanup completed', + ); + }); + }); + + describe('dismissed auto-suggestion cleanup', () => { + it('should delete old dismissed associations using span retention cutoff', () => { + mockAppSettingsGet.mockReturnValue('5'); + mockDeleteOldDismissed.mockReturnValue(3); + + const now = new Date('2026-02-21T10:00:00Z'); + jest.setSystemTime(now); + + const service = DataRetentionService.getInstance(); + const result = service.runCleanup(); + + expect(result.dismissedAssociationsDeleted).toBe(3); + expect(mockDeleteOldDismissed).toHaveBeenCalledWith(expect.stringMatching(/^\d{4}-\d{2}-\d{2}T/)); + + // Should use same cutoff as spans (5 days) + const spanCutoff = mockDeleteSpans.mock.calls[0][0]; + const dismissedCutoff = mockDeleteOldDismissed.mock.calls[0][0]; + expect(spanCutoff).toBe(dismissedCutoff); + }); + + it('should include dismissedAssociationsDeleted in log output', () => { + mockDeleteOldDismissed.mockReturnValue(7); + + const service = DataRetentionService.getInstance(); + service.runCleanup(); + + expect(logger.info).toHaveBeenCalledWith( + expect.objectContaining({ dismissedAssociationsDeleted: 7 }), + 'data retention cleanup completed', + ); + }); + + it('should report zero when no dismissed associations to clean', () => { + mockDeleteOldDismissed.mockReturnValue(0); + + const service = DataRetentionService.getInstance(); + const result = service.runCleanup(); + + expect(result.dismissedAssociationsDeleted).toBe(0); + }); + }); + describe('scheduled cleanup', () => { it('should run cleanup when past the scheduled time on startup', () => { // Set time after cleanup time (02:00) diff --git a/server/src/services/retention/DataRetentionService.ts b/server/src/services/retention/DataRetentionService.ts index 936454b..51cf50c 100644 --- a/server/src/services/retention/DataRetentionService.ts +++ b/server/src/services/retention/DataRetentionService.ts @@ -138,6 +138,29 @@ export class DataRetentionService { // Clean up expired alert mutes (not retention-based — they self-expire) const mutesExpired = stores.alertMutes.deleteExpired(); + // Span retention: uses configurable span_retention_days from app_settings (default 7) + const spanRetentionDaysStr = stores.appSettings.get('span_retention_days'); + const spanRetentionDays = spanRetentionDaysStr ? parseInt(spanRetentionDaysStr, 10) : 7; + const spanCutoff = new Date(); + spanCutoff.setDate(spanCutoff.getDate() - spanRetentionDays); + const spanCutoffTimestamp = spanCutoff.toISOString(); + + const spansDeleted = stores.spans.deleteOlderThan(spanCutoffTimestamp); + + // Dismissed auto-suggestion cleanup: uses same span retention window + const dismissedAssociationsDeleted = stores.associations.deleteOldDismissed(spanCutoffTimestamp); + + // Usage bucket retention: minute=24h, hour=30d, orphaned=7d + const usageMinuteDeleted = stores.apiKeyUsage.pruneMinuteBuckets( + new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(), + ); + const usageHourDeleted = stores.apiKeyUsage.pruneHourBuckets( + new Date(Date.now() - 30 * 24 * 60 * 60 * 1000).toISOString(), + ); + const usageOrphanedDeleted = stores.apiKeyUsage.pruneOrphanedBuckets( + new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString(), + ); + const result: CleanupResult = { latencyDeleted, errorDeleted, @@ -148,6 +171,11 @@ export class DataRetentionService { syncHistoryDeleted, driftFlagsDeleted, mutesExpired, + usageMinuteDeleted, + usageHourDeleted, + usageOrphanedDeleted, + spansDeleted, + dismissedAssociationsDeleted, retentionDays, cutoffTimestamp, }; @@ -201,6 +229,11 @@ export interface CleanupResult { syncHistoryDeleted: number; driftFlagsDeleted: number; mutesExpired: number; + usageMinuteDeleted: number; + usageHourDeleted: number; + usageOrphanedDeleted: number; + spansDeleted: number; + dismissedAssociationsDeleted: number; retentionDays: number; cutoffTimestamp: string; } diff --git a/server/src/services/wallboard/WallboardService.test.ts b/server/src/services/wallboard/WallboardService.test.ts index 2cf6c75..61b4bfb 100644 --- a/server/src/services/wallboard/WallboardService.test.ts +++ b/server/src/services/wallboard/WallboardService.test.ts @@ -35,6 +35,10 @@ function makeDep(overrides: Partial = {}): DependencyFor check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: '2025-01-01T12:00:00Z', last_status_change: null, @@ -44,8 +48,10 @@ function makeDep(overrides: Partial = {}): DependencyFor service_team_id: 'team-1', service_team_name: 'Team One', target_service_id: null, + association_id: null, association_type: null, avg_latency_24h: null, + is_auto_suggested: null, linked_service_name: null, ...overrides, }; diff --git a/server/src/stores/impl/ApiKeyUsageStore.test.ts b/server/src/stores/impl/ApiKeyUsageStore.test.ts new file mode 100644 index 0000000..2b71efa --- /dev/null +++ b/server/src/stores/impl/ApiKeyUsageStore.test.ts @@ -0,0 +1,270 @@ +import Database from 'better-sqlite3'; +import { ApiKeyUsageBucket } from '../../db/types'; +import { ApiKeyUsageStore } from './ApiKeyUsageStore'; + +describe('ApiKeyUsageStore', () => { + let db: Database.Database; + let store: ApiKeyUsageStore; + + beforeEach(() => { + db = new Database(':memory:'); + db.exec(` + CREATE TABLE api_key_usage_buckets ( + api_key_id TEXT NOT NULL, + bucket_start TEXT NOT NULL, + granularity TEXT NOT NULL CHECK(granularity IN ('minute', 'hour')), + push_count INTEGER NOT NULL DEFAULT 0, + rejected_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (api_key_id, bucket_start, granularity) + ); + CREATE INDEX idx_usage_buckets_key_start ON api_key_usage_buckets(api_key_id, bucket_start); + CREATE INDEX idx_usage_buckets_start ON api_key_usage_buckets(bucket_start); + + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT + ); + `); + store = new ApiKeyUsageStore(db); + }); + + afterEach(() => { + db.close(); + }); + + describe('bulkUpsert (DPS-100j)', () => { + it('should insert new rows correctly', () => { + store.bulkUpsert([ + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:32:00', granularity: 'minute', push_count: 5, rejected_count: 1 }, + ]); + + const rows = db.prepare('SELECT * FROM api_key_usage_buckets').all() as ApiKeyUsageBucket[]; + expect(rows).toHaveLength(1); + expect(rows[0].api_key_id).toBe('key-1'); + expect(rows[0].bucket_start).toBe('2025-01-15T14:32:00'); + expect(rows[0].granularity).toBe('minute'); + expect(rows[0].push_count).toBe(5); + expect(rows[0].rejected_count).toBe(1); + }); + + it('should accumulate push_count and rejected_count on conflict', () => { + store.bulkUpsert([ + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:32:00', granularity: 'minute', push_count: 5, rejected_count: 1 }, + ]); + + // Upsert same bucket again + store.bulkUpsert([ + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:32:00', granularity: 'minute', push_count: 3, rejected_count: 2 }, + ]); + + const rows = db.prepare('SELECT * FROM api_key_usage_buckets').all() as ApiKeyUsageBucket[]; + expect(rows).toHaveLength(1); + expect(rows[0].push_count).toBe(8); // 5 + 3 + expect(rows[0].rejected_count).toBe(3); // 1 + 2 + }); + + it('should handle mixed new and existing entries in one transaction', () => { + store.bulkUpsert([ + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:32:00', granularity: 'minute', push_count: 5, rejected_count: 0 }, + ]); + + store.bulkUpsert([ + // Existing row — should accumulate + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:32:00', granularity: 'minute', push_count: 3, rejected_count: 0 }, + // New row + { api_key_id: 'key-2', bucket_start: '2025-01-15T14:32:00', granularity: 'minute', push_count: 10, rejected_count: 0 }, + ]); + + const rows = db.prepare('SELECT * FROM api_key_usage_buckets ORDER BY api_key_id').all() as ApiKeyUsageBucket[]; + expect(rows).toHaveLength(2); + expect(rows[0].api_key_id).toBe('key-1'); + expect(rows[0].push_count).toBe(8); // 5 + 3 + expect(rows[1].api_key_id).toBe('key-2'); + expect(rows[1].push_count).toBe(10); + }); + + it('should do nothing when entries is empty', () => { + store.bulkUpsert([]); + + const rows = db.prepare('SELECT * FROM api_key_usage_buckets').all(); + expect(rows).toHaveLength(0); + }); + }); + + describe('getBuckets (DPS-100k)', () => { + beforeEach(() => { + store.bulkUpsert([ + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:00:00', granularity: 'minute', push_count: 10, rejected_count: 0 }, + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:01:00', granularity: 'minute', push_count: 20, rejected_count: 1 }, + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:02:00', granularity: 'minute', push_count: 30, rejected_count: 0 }, + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:00:00', granularity: 'hour', push_count: 100, rejected_count: 5 }, + { api_key_id: 'key-1', bucket_start: '2025-01-15T15:00:00', granularity: 'hour', push_count: 200, rejected_count: 0 }, + { api_key_id: 'key-2', bucket_start: '2025-01-15T14:00:00', granularity: 'minute', push_count: 50, rejected_count: 0 }, + ]); + }); + + it('should return rows in the specified time range ordered by bucket_start ASC', () => { + const buckets = store.getBuckets('key-1', 'minute', '2025-01-15T14:00:00', '2025-01-15T14:01:00'); + + expect(buckets).toHaveLength(2); + expect(buckets[0].bucket_start).toBe('2025-01-15T14:00:00'); + expect(buckets[1].bucket_start).toBe('2025-01-15T14:01:00'); + }); + + it('should return only rows matching the specified granularity', () => { + const minuteBuckets = store.getBuckets('key-1', 'minute', '2025-01-15T14:00:00', '2025-01-15T15:00:00'); + const hourBuckets = store.getBuckets('key-1', 'hour', '2025-01-15T14:00:00', '2025-01-15T15:00:00'); + + expect(minuteBuckets).toHaveLength(3); + expect(minuteBuckets.every(b => b.granularity === 'minute')).toBe(true); + + expect(hourBuckets).toHaveLength(2); + expect(hourBuckets.every(b => b.granularity === 'hour')).toBe(true); + }); + + it('should return empty array for a key with no data', () => { + const buckets = store.getBuckets('key-nonexistent', 'minute', '2025-01-15T14:00:00', '2025-01-15T15:00:00'); + expect(buckets).toEqual([]); + }); + + it('should not return rows outside the time range', () => { + const buckets = store.getBuckets('key-1', 'minute', '2025-01-15T14:00:00', '2025-01-15T14:00:00'); + expect(buckets).toHaveLength(1); + expect(buckets[0].bucket_start).toBe('2025-01-15T14:00:00'); + }); + }); + + describe('getSummaryForKeys (DPS-100k)', () => { + beforeEach(() => { + store.bulkUpsert([ + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:00:00', granularity: 'minute', push_count: 10, rejected_count: 1 }, + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:01:00', granularity: 'minute', push_count: 20, rejected_count: 2 }, + { api_key_id: 'key-2', bucket_start: '2025-01-15T14:00:00', granularity: 'minute', push_count: 50, rejected_count: 5 }, + ]); + }); + + it('should aggregate push_count and rejected_count correctly across rows', () => { + const summary = store.getSummaryForKeys(['key-1'], '2025-01-15T14:00:00', '2025-01-15T14:01:00'); + + const key1 = summary.get('key-1'); + expect(key1).toBeDefined(); + expect(key1!.push_count).toBe(30); // 10 + 20 + expect(key1!.rejected_count).toBe(3); // 1 + 2 + }); + + it('should handle multiple keys', () => { + const summary = store.getSummaryForKeys(['key-1', 'key-2'], '2025-01-15T14:00:00', '2025-01-15T14:01:00'); + + expect(summary.get('key-1')!.push_count).toBe(30); + expect(summary.get('key-2')!.push_count).toBe(50); + }); + + it('should not include keys that have no data in the range', () => { + const summary = store.getSummaryForKeys(['key-1', 'key-nonexistent'], '2025-01-15T14:00:00', '2025-01-15T14:01:00'); + + expect(summary.has('key-1')).toBe(true); + expect(summary.has('key-nonexistent')).toBe(false); + }); + + it('should return empty map for empty key IDs array', () => { + const summary = store.getSummaryForKeys([], '2025-01-15T14:00:00', '2025-01-15T14:01:00'); + expect(summary.size).toBe(0); + }); + }); + + describe('prune methods (DPS-100l)', () => { + beforeEach(() => { + store.bulkUpsert([ + // Old minute rows + { api_key_id: 'key-1', bucket_start: '2025-01-14T10:00:00', granularity: 'minute', push_count: 5, rejected_count: 0 }, + { api_key_id: 'key-1', bucket_start: '2025-01-14T10:01:00', granularity: 'minute', push_count: 5, rejected_count: 0 }, + // Recent minute row + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:00:00', granularity: 'minute', push_count: 10, rejected_count: 0 }, + // Old hour rows + { api_key_id: 'key-1', bucket_start: '2025-01-14T10:00:00', granularity: 'hour', push_count: 100, rejected_count: 0 }, + // Recent hour row + { api_key_id: 'key-1', bucket_start: '2025-01-15T14:00:00', granularity: 'hour', push_count: 200, rejected_count: 0 }, + ]); + }); + + describe('pruneMinuteBuckets', () => { + it('should delete minute rows older than cutoff', () => { + const deleted = store.pruneMinuteBuckets('2025-01-15T00:00:00'); + + expect(deleted).toBe(2); // two old minute rows + + const remaining = db.prepare('SELECT * FROM api_key_usage_buckets WHERE granularity = ?').all('minute') as ApiKeyUsageBucket[]; + expect(remaining).toHaveLength(1); + expect(remaining[0].bucket_start).toBe('2025-01-15T14:00:00'); + }); + + it('should not delete hour rows', () => { + store.pruneMinuteBuckets('2025-01-15T00:00:00'); + + const hourRows = db.prepare('SELECT * FROM api_key_usage_buckets WHERE granularity = ?').all('hour') as ApiKeyUsageBucket[]; + expect(hourRows).toHaveLength(2); + }); + }); + + describe('pruneHourBuckets', () => { + it('should delete hour rows older than cutoff', () => { + const deleted = store.pruneHourBuckets('2025-01-15T00:00:00'); + + expect(deleted).toBe(1); // one old hour row + + const remaining = db.prepare('SELECT * FROM api_key_usage_buckets WHERE granularity = ?').all('hour') as ApiKeyUsageBucket[]; + expect(remaining).toHaveLength(1); + expect(remaining[0].bucket_start).toBe('2025-01-15T14:00:00'); + }); + + it('should not delete minute rows', () => { + store.pruneHourBuckets('2025-01-15T00:00:00'); + + const minuteRows = db.prepare('SELECT * FROM api_key_usage_buckets WHERE granularity = ?').all('minute') as ApiKeyUsageBucket[]; + expect(minuteRows).toHaveLength(3); + }); + }); + + describe('pruneOrphanedBuckets', () => { + it('should delete rows for keys not in team_api_keys table after grace period', () => { + // key-1 has no entry in team_api_keys -> orphaned + // Old rows should be pruned + const deleted = store.pruneOrphanedBuckets('2025-01-15T00:00:00'); + + // Rows older than cutoff and orphaned are deleted + expect(deleted).toBeGreaterThan(0); + }); + + it('should retain rows for keys that exist in team_api_keys', () => { + // Insert a team_api_keys entry for key-1 + db.prepare(` + INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix) + VALUES ('key-1', 'team-1', 'Test', 'hash1', 'dps_test') + `).run(); + + const deleted = store.pruneOrphanedBuckets('2025-01-15T00:00:00'); + + expect(deleted).toBe(0); + + const rows = db.prepare('SELECT * FROM api_key_usage_buckets').all(); + expect(rows).toHaveLength(5); // all rows retained + }); + + it('should retain orphaned rows within the grace period', () => { + // All rows have bucket_start >= '2025-01-14T10:00:00' + // Using a cutoff before all rows means nothing is pruned + const deleted = store.pruneOrphanedBuckets('2025-01-14T09:00:00'); + + expect(deleted).toBe(0); + }); + }); + }); +}); diff --git a/server/src/stores/impl/ApiKeyUsageStore.ts b/server/src/stores/impl/ApiKeyUsageStore.ts new file mode 100644 index 0000000..cc59604 --- /dev/null +++ b/server/src/stores/impl/ApiKeyUsageStore.ts @@ -0,0 +1,124 @@ +import { Database } from 'better-sqlite3'; +import { ApiKeyUsageBucket } from '../../db/types'; +import { IApiKeyUsageStore, BulkUpsertEntry } from '../interfaces/IApiKeyUsageStore'; + +export class ApiKeyUsageStore implements IApiKeyUsageStore { + constructor(private db: Database) {} + + bulkUpsert(entries: BulkUpsertEntry[]): void { + if (entries.length === 0) return; + + const stmt = this.db.prepare( + `INSERT INTO api_key_usage_buckets (api_key_id, bucket_start, granularity, push_count, rejected_count) + VALUES (?, ?, ?, ?, ?) + ON CONFLICT(api_key_id, bucket_start, granularity) + DO UPDATE SET + push_count = push_count + excluded.push_count, + rejected_count = rejected_count + excluded.rejected_count`, + ); + + const runAll = this.db.transaction((rows: BulkUpsertEntry[]) => { + for (const row of rows) { + stmt.run(row.api_key_id, row.bucket_start, row.granularity, row.push_count, row.rejected_count); + } + }); + + runAll(entries); + } + + getBuckets(apiKeyId: string, granularity: 'minute' | 'hour', from: string, to: string): ApiKeyUsageBucket[] { + return this.db + .prepare( + `SELECT * FROM api_key_usage_buckets + WHERE api_key_id = ? AND granularity = ? + AND bucket_start >= ? AND bucket_start <= ? + ORDER BY bucket_start ASC`, + ) + .all(apiKeyId, granularity, from, to) as ApiKeyUsageBucket[]; + } + + getBucketsByTeam( + teamId: string, + granularity: 'minute' | 'hour', + from: string, + to: string, + ): (ApiKeyUsageBucket & { key_name: string; key_prefix: string })[] { + return this.db + .prepare( + `SELECT b.*, k.name AS key_name, k.key_prefix + FROM api_key_usage_buckets b + JOIN team_api_keys k ON k.id = b.api_key_id + WHERE k.team_id = ? AND b.granularity = ? + AND b.bucket_start >= ? AND b.bucket_start <= ? + ORDER BY b.bucket_start ASC`, + ) + .all(teamId, granularity, from, to) as (ApiKeyUsageBucket & { key_name: string; key_prefix: string })[]; + } + + getAllBuckets( + granularity: 'minute' | 'hour', + from: string, + to: string, + ): (ApiKeyUsageBucket & { team_id: string; key_name: string })[] { + return this.db + .prepare( + `SELECT b.*, k.team_id, k.name AS key_name + FROM api_key_usage_buckets b + JOIN team_api_keys k ON k.id = b.api_key_id + WHERE b.granularity = ? + AND b.bucket_start >= ? AND b.bucket_start <= ? + ORDER BY b.bucket_start ASC`, + ) + .all(granularity, from, to) as (ApiKeyUsageBucket & { team_id: string; key_name: string })[]; + } + + getSummaryForKeys( + apiKeyIds: string[], + from: string, + to: string, + ): Map { + const result = new Map(); + if (apiKeyIds.length === 0) return result; + + const placeholders = apiKeyIds.map(() => '?').join(', '); + const rows = this.db + .prepare( + `SELECT api_key_id, + SUM(push_count) AS push_count, + SUM(rejected_count) AS rejected_count + FROM api_key_usage_buckets + WHERE api_key_id IN (${placeholders}) + AND bucket_start >= ? AND bucket_start <= ? + GROUP BY api_key_id`, + ) + .all(...apiKeyIds, from, to) as { api_key_id: string; push_count: number; rejected_count: number }[]; + + for (const row of rows) { + result.set(row.api_key_id, { push_count: row.push_count, rejected_count: row.rejected_count }); + } + + return result; + } + + pruneMinuteBuckets(olderThan: string): number { + return this.db + .prepare(`DELETE FROM api_key_usage_buckets WHERE granularity = 'minute' AND bucket_start < ?`) + .run(olderThan).changes; + } + + pruneHourBuckets(olderThan: string): number { + return this.db + .prepare(`DELETE FROM api_key_usage_buckets WHERE granularity = 'hour' AND bucket_start < ?`) + .run(olderThan).changes; + } + + pruneOrphanedBuckets(olderThan: string): number { + return this.db + .prepare( + `DELETE FROM api_key_usage_buckets + WHERE api_key_id NOT IN (SELECT id FROM team_api_keys) + AND bucket_start < ?`, + ) + .run(olderThan).changes; + } +} diff --git a/server/src/stores/impl/AppSettingsStore.test.ts b/server/src/stores/impl/AppSettingsStore.test.ts new file mode 100644 index 0000000..11d58f7 --- /dev/null +++ b/server/src/stores/impl/AppSettingsStore.test.ts @@ -0,0 +1,49 @@ +import Database from 'better-sqlite3'; +import { runMigrations } from '../../db/migrate'; +import { AppSettingsStore } from './AppSettingsStore'; + +describe('AppSettingsStore', () => { + let db: Database.Database; + let store: AppSettingsStore; + + beforeEach(() => { + db = new Database(':memory:'); + db.pragma('foreign_keys = ON'); + runMigrations(db); + store = new AppSettingsStore(db); + }); + + afterEach(() => { + db.close(); + }); + + describe('get', () => { + it('returns seeded span_retention_days value', () => { + expect(store.get('span_retention_days')).toBe('7'); + }); + + it('returns undefined for missing key', () => { + expect(store.get('nonexistent_key')).toBeUndefined(); + }); + }); + + describe('set', () => { + it('creates a new entry', () => { + store.set('custom_key', 'custom_value'); + expect(store.get('custom_key')).toBe('custom_value'); + }); + + it('updates an existing entry', () => { + store.set('span_retention_days', '14'); + expect(store.get('span_retention_days')).toBe('14'); + }); + + it('stores updatedBy when provided', () => { + db.prepare("INSERT INTO users (id, email, name, role) VALUES ('u1', 'a@b.com', 'Admin', 'admin')").run(); + store.set('span_retention_days', '30', 'u1'); + + const row = db.prepare("SELECT updated_by FROM app_settings WHERE key = 'span_retention_days'").get() as { updated_by: string }; + expect(row.updated_by).toBe('u1'); + }); + }); +}); diff --git a/server/src/stores/impl/AppSettingsStore.ts b/server/src/stores/impl/AppSettingsStore.ts new file mode 100644 index 0000000..2df473e --- /dev/null +++ b/server/src/stores/impl/AppSettingsStore.ts @@ -0,0 +1,26 @@ +import { Database } from 'better-sqlite3'; +import { IAppSettingsStore } from '../interfaces/IAppSettingsStore'; + +export class AppSettingsStore implements IAppSettingsStore { + constructor(private db: Database) {} + + get(key: string): string | undefined { + const row = this.db + .prepare('SELECT value FROM app_settings WHERE key = ?') + .get(key) as { value: string } | undefined; + return row?.value; + } + + set(key: string, value: string, updatedBy?: string): void { + this.db + .prepare(` + INSERT INTO app_settings (key, value, updated_at, updated_by) + VALUES (?, ?, datetime('now'), ?) + ON CONFLICT(key) DO UPDATE SET + value = excluded.value, + updated_at = excluded.updated_at, + updated_by = excluded.updated_by + `) + .run(key, value, updatedBy ?? null); + } +} diff --git a/server/src/stores/impl/AssociationStore.test.ts b/server/src/stores/impl/AssociationStore.test.ts index 523a9f2..21d1138 100644 --- a/server/src/stores/impl/AssociationStore.test.ts +++ b/server/src/stores/impl/AssociationStore.test.ts @@ -27,7 +27,8 @@ describe('AssociationStore', () => { id TEXT PRIMARY KEY, service_id TEXT NOT NULL, name TEXT NOT NULL, - skipped INTEGER NOT NULL DEFAULT 0 + skipped INTEGER NOT NULL DEFAULT 0, + discovery_source TEXT NOT NULL DEFAULT 'manual' ); CREATE TABLE dependency_associations ( @@ -35,6 +36,8 @@ describe('AssociationStore', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), UNIQUE (dependency_id, linked_service_id) @@ -217,4 +220,77 @@ describe('AssociationStore', () => { expect(store.count({ linkedServiceId: testLinkedServiceId })).toBe(1); }); }); + + describe('auto-suggestion methods', () => { + it('create with is_auto_suggested=true sets the flag', () => { + const assoc = store.create({ + dependency_id: testDependencyId, + linked_service_id: 'svc-789', + association_type: 'api_call', + is_auto_suggested: true, + }); + + expect(assoc.is_auto_suggested).toBe(1); + expect(assoc.is_dismissed).toBe(0); + }); + + it('create without is_auto_suggested defaults to 0', () => { + const assoc = store.create({ + dependency_id: testDependencyId, + linked_service_id: 'svc-789', + association_type: 'database', + }); + + expect(assoc.is_auto_suggested).toBe(0); + }); + + it('findAutoSuggested returns only auto-suggested non-dismissed', () => { + // Auto-suggested, not dismissed + store.create({ + dependency_id: testDependencyId, + linked_service_id: 'svc-789', + association_type: 'api_call', + is_auto_suggested: true, + }); + + // Regular (not auto-suggested) — already exists from beforeEach + const autoSuggested = store.findAutoSuggested(testDependencyId); + expect(autoSuggested).toHaveLength(1); + expect(autoSuggested[0].is_auto_suggested).toBe(1); + }); + + it('confirm sets is_auto_suggested=0', () => { + const assoc = store.create({ + dependency_id: testDependencyId, + linked_service_id: 'svc-789', + association_type: 'api_call', + is_auto_suggested: true, + }); + + expect(store.confirm(assoc.id)).toBe(true); + + const updated = store.findById(assoc.id)!; + expect(updated.is_auto_suggested).toBe(0); + + // Should no longer appear in findAutoSuggested + expect(store.findAutoSuggested(testDependencyId)).toHaveLength(0); + }); + + it('dismiss sets is_dismissed=1', () => { + const assoc = store.create({ + dependency_id: testDependencyId, + linked_service_id: 'svc-789', + association_type: 'api_call', + is_auto_suggested: true, + }); + + expect(store.dismiss(assoc.id)).toBe(true); + + const updated = store.findById(assoc.id)!; + expect(updated.is_dismissed).toBe(1); + + // Should no longer appear in findAutoSuggested + expect(store.findAutoSuggested(testDependencyId)).toHaveLength(0); + }); + }); }); diff --git a/server/src/stores/impl/AssociationStore.ts b/server/src/stores/impl/AssociationStore.ts index 28f12c9..d57c54a 100644 --- a/server/src/stores/impl/AssociationStore.ts +++ b/server/src/stores/impl/AssociationStore.ts @@ -55,27 +55,58 @@ export class AssociationStore implements IAssociationStore { return row !== undefined; } + findAutoSuggested(dependencyId: string): DependencyAssociation[] { + return this.db + .prepare( + 'SELECT * FROM dependency_associations WHERE dependency_id = ? AND is_auto_suggested = 1 AND is_dismissed = 0' + ) + .all(dependencyId) as DependencyAssociation[]; + } + create(input: AssociationCreateInput): DependencyAssociation { const id = randomUUID(); const now = new Date().toISOString(); + const isAutoSuggested = input.is_auto_suggested ? 1 : 0; this.db .prepare(` INSERT INTO dependency_associations ( - id, dependency_id, linked_service_id, association_type, manifest_managed, created_at - ) VALUES (?, ?, ?, ?, 0, ?) + id, dependency_id, linked_service_id, association_type, is_auto_suggested, manifest_managed, created_at + ) VALUES (?, ?, ?, ?, ?, 0, ?) `) .run( id, input.dependency_id, input.linked_service_id, input.association_type, + isAutoSuggested, now ); return this.findById(id)!; } + confirm(id: string): boolean { + const result = this.db + .prepare('UPDATE dependency_associations SET is_auto_suggested = 0 WHERE id = ?') + .run(id); + return result.changes > 0; + } + + dismiss(id: string): boolean { + const result = this.db + .prepare('UPDATE dependency_associations SET is_dismissed = 1 WHERE id = ?') + .run(id); + return result.changes > 0; + } + + deleteOldDismissed(olderThan: string): number { + const result = this.db + .prepare('DELETE FROM dependency_associations WHERE is_dismissed = 1 AND created_at < ?') + .run(olderThan); + return result.changes; + } + delete(id: string): boolean { const result = this.db .prepare('DELETE FROM dependency_associations WHERE id = ?') diff --git a/server/src/stores/impl/DependencyStore.test.ts b/server/src/stores/impl/DependencyStore.test.ts index 66abe2d..4700d12 100644 --- a/server/src/stores/impl/DependencyStore.test.ts +++ b/server/src/stores/impl/DependencyStore.test.ts @@ -48,6 +48,10 @@ describe('DependencyStore', () => { error TEXT, error_message TEXT, skipped INTEGER NOT NULL DEFAULT 0, + discovery_source TEXT NOT NULL DEFAULT 'manual', + user_display_name TEXT, + user_description TEXT, + user_impact TEXT, last_checked TEXT, last_status_change TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), @@ -60,6 +64,8 @@ describe('DependencyStore', () => { dependency_id TEXT NOT NULL, linked_service_id TEXT NOT NULL, association_type TEXT DEFAULT 'api_call', + is_auto_suggested INTEGER NOT NULL DEFAULT 0, + is_dismissed INTEGER NOT NULL DEFAULT 0, manifest_managed INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), FOREIGN KEY (dependency_id) REFERENCES dependencies(id) ON DELETE CASCADE @@ -1003,4 +1009,93 @@ describe('DependencyStore', () => { expect(results.some((d) => d.name === 'ActiveDep')).toBe(true); }); }); + + describe('upsert discovery_source', () => { + it('should pass through discovery_source on insert', () => { + const result = store.upsert({ + service_id: testServiceId, + name: 'TraceDep', + healthy: true, + health_state: 0, + health_code: 200, + latency_ms: 50, + last_checked: new Date().toISOString(), + discovery_source: 'otlp_trace', + }); + + expect(result.dependency.discovery_source).toBe('otlp_trace'); + }); + + it('should default discovery_source to manual', () => { + const result = store.upsert({ + service_id: testServiceId, + name: 'ManualDep', + healthy: true, + health_state: 0, + health_code: 200, + latency_ms: 50, + last_checked: new Date().toISOString(), + }); + + expect(result.dependency.discovery_source).toBe('manual'); + }); + + it('should preserve manual source on conflict with otlp_trace', () => { + const now = new Date().toISOString(); + + // First insert as manual + store.upsert({ + service_id: testServiceId, + name: 'PreserveDep', + healthy: true, + health_state: 0, + health_code: 200, + latency_ms: 50, + last_checked: now, + discovery_source: 'manual', + }); + + // Second upsert as otlp_trace — should NOT overwrite manual + const result = store.upsert({ + service_id: testServiceId, + name: 'PreserveDep', + healthy: true, + health_state: 0, + health_code: 200, + latency_ms: 60, + last_checked: now, + discovery_source: 'otlp_trace', + }); + + expect(result.dependency.discovery_source).toBe('manual'); + }); + + it('should allow upgrade from otlp_metric to otlp_trace', () => { + const now = new Date().toISOString(); + + store.upsert({ + service_id: testServiceId, + name: 'UpgradeDep', + healthy: true, + health_state: 0, + health_code: 200, + latency_ms: 50, + last_checked: now, + discovery_source: 'otlp_metric', + }); + + const result = store.upsert({ + service_id: testServiceId, + name: 'UpgradeDep', + healthy: true, + health_state: 0, + health_code: 200, + latency_ms: 60, + last_checked: now, + discovery_source: 'otlp_trace', + }); + + expect(result.dependency.discovery_source).toBe('otlp_trace'); + }); + }); }); diff --git a/server/src/stores/impl/DependencyStore.ts b/server/src/stores/impl/DependencyStore.ts index 4c88753..917b4a8 100644 --- a/server/src/stores/impl/DependencyStore.ts +++ b/server/src/stores/impl/DependencyStore.ts @@ -12,6 +12,7 @@ import { DependencyListOptions, DependencyUpsertInput, DependencyOverrideInput, + DependencyUserEnrichmentInput, DependentReport, } from '../types'; import { validateOrderBy } from '../orderByValidator'; @@ -47,7 +48,9 @@ export class DependencyStore implements IDependencyStore { d.*, s.name as service_name, da.linked_service_id as target_service_id, + da.id as association_id, da.association_type, + da.is_auto_suggested, ( SELECT ROUND(AVG(latency_ms)) FROM dependency_latency_history @@ -93,7 +96,9 @@ export class DependencyStore implements IDependencyStore { d.error_message, s.name as service_name, da.linked_service_id as target_service_id, + da.id as association_id, da.association_type, + da.is_auto_suggested, ( SELECT ROUND(AVG(latency_ms)) FROM dependency_latency_history @@ -124,7 +129,9 @@ export class DependencyStore implements IDependencyStore { d.error_message, s.name as service_name, da.linked_service_id as target_service_id, + da.id as association_id, da.association_type, + da.is_auto_suggested, ( SELECT ROUND(AVG(latency_ms)) FROM dependency_latency_history @@ -148,7 +155,9 @@ export class DependencyStore implements IDependencyStore { s.team_id as service_team_id, t.name as service_team_name, da.linked_service_id as target_service_id, + da.id as association_id, da.association_type, + da.is_auto_suggested, ls.name as linked_service_name, ( SELECT ROUND(AVG(latency_ms)) @@ -221,8 +230,9 @@ export class DependencyStore implements IDependencyStore { id, service_id, name, canonical_name, description, impact, type, healthy, health_state, health_code, latency_ms, contact, check_details, error, error_message, skipped, + discovery_source, last_checked, last_status_change, created_at, updated_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(service_id, name) DO UPDATE SET canonical_name = excluded.canonical_name, description = excluded.description, @@ -237,6 +247,10 @@ export class DependencyStore implements IDependencyStore { error = excluded.error, error_message = excluded.error_message, skipped = excluded.skipped, + discovery_source = CASE + WHEN dependencies.discovery_source = 'manual' THEN dependencies.discovery_source + ELSE excluded.discovery_source + END, last_checked = excluded.last_checked, last_status_change = CASE WHEN dependencies.healthy IS NULL OR dependencies.healthy != excluded.healthy @@ -262,6 +276,7 @@ export class DependencyStore implements IDependencyStore { errorJson, input.error_message ?? null, skippedValue, + input.discovery_source ?? 'manual', input.last_checked, now, now, @@ -304,6 +319,43 @@ export class DependencyStore implements IDependencyStore { return this.findById(id)!; } + findByDiscoverySource(serviceId: string, source: string): Dependency[] { + return this.db + .prepare('SELECT * FROM dependencies WHERE service_id = ? AND discovery_source = ? ORDER BY name ASC') + .all(serviceId, source) as Dependency[]; + } + + updateUserEnrichment(id: string, enrichment: DependencyUserEnrichmentInput): Dependency | undefined { + const existing = this.findById(id); + if (!existing) return undefined; + + const setClauses: string[] = ['updated_at = ?']; + const params: unknown[] = [new Date().toISOString()]; + + if ('displayName' in enrichment) { + setClauses.push('user_display_name = ?'); + params.push(enrichment.displayName ?? null); + } + + if ('description' in enrichment) { + setClauses.push('user_description = ?'); + params.push(enrichment.description ?? null); + } + + if ('impact' in enrichment) { + setClauses.push('user_impact = ?'); + params.push(enrichment.impact ?? null); + } + + params.push(id); + + this.db + .prepare(`UPDATE dependencies SET ${setClauses.join(', ')} WHERE id = ?`) + .run(...params); + + return this.findById(id)!; + } + delete(id: string): boolean { const result = this.db .prepare('DELETE FROM dependencies WHERE id = ?') diff --git a/server/src/stores/impl/ExternalNodeEnrichmentStore.test.ts b/server/src/stores/impl/ExternalNodeEnrichmentStore.test.ts new file mode 100644 index 0000000..d6352b4 --- /dev/null +++ b/server/src/stores/impl/ExternalNodeEnrichmentStore.test.ts @@ -0,0 +1,85 @@ +import Database from 'better-sqlite3'; +import { runMigrations } from '../../db/migrate'; +import { ExternalNodeEnrichmentStore } from './ExternalNodeEnrichmentStore'; + +describe('ExternalNodeEnrichmentStore', () => { + let db: Database.Database; + let store: ExternalNodeEnrichmentStore; + + beforeEach(() => { + db = new Database(':memory:'); + db.pragma('foreign_keys = ON'); + runMigrations(db); + store = new ExternalNodeEnrichmentStore(db); + }); + + afterEach(() => { + db.close(); + }); + + describe('upsert', () => { + it('creates a new record', () => { + const result = store.upsert({ + canonical_name: 'PostgreSQL', + display_name: 'PostgreSQL Database', + description: 'Primary relational database', + impact: 'Critical', + service_type: 'database', + }); + + expect(result.canonical_name).toBe('PostgreSQL'); + expect(result.display_name).toBe('PostgreSQL Database'); + expect(result.description).toBe('Primary relational database'); + expect(result.impact).toBe('Critical'); + expect(result.service_type).toBe('database'); + expect(result.id).toBeDefined(); + }); + + it('updates existing record by canonical_name', () => { + store.upsert({ canonical_name: 'Redis', display_name: 'Redis Cache' }); + const updated = store.upsert({ canonical_name: 'Redis', display_name: 'Redis Session Store', impact: 'High' }); + + expect(updated.display_name).toBe('Redis Session Store'); + expect(updated.impact).toBe('High'); + }); + }); + + describe('findByCanonicalName', () => { + it('returns matching record', () => { + store.upsert({ canonical_name: 'Kafka' }); + const result = store.findByCanonicalName('Kafka'); + expect(result).toBeDefined(); + expect(result!.canonical_name).toBe('Kafka'); + }); + + it('returns undefined for non-existent name', () => { + expect(store.findByCanonicalName('nonexistent')).toBeUndefined(); + }); + }); + + describe('findAll', () => { + it('returns all records ordered by canonical_name', () => { + store.upsert({ canonical_name: 'Zebra' }); + store.upsert({ canonical_name: 'Apple' }); + store.upsert({ canonical_name: 'Mango' }); + + const all = store.findAll(); + expect(all).toHaveLength(3); + expect(all[0].canonical_name).toBe('Apple'); + expect(all[1].canonical_name).toBe('Mango'); + expect(all[2].canonical_name).toBe('Zebra'); + }); + }); + + describe('delete', () => { + it('removes the record', () => { + const record = store.upsert({ canonical_name: 'ToDelete' }); + expect(store.delete(record.id)).toBe(true); + expect(store.findByCanonicalName('ToDelete')).toBeUndefined(); + }); + + it('returns false for non-existent id', () => { + expect(store.delete('nonexistent')).toBe(false); + }); + }); +}); diff --git a/server/src/stores/impl/ExternalNodeEnrichmentStore.ts b/server/src/stores/impl/ExternalNodeEnrichmentStore.ts new file mode 100644 index 0000000..d2cfaa2 --- /dev/null +++ b/server/src/stores/impl/ExternalNodeEnrichmentStore.ts @@ -0,0 +1,58 @@ +import { randomUUID } from 'crypto'; +import { Database } from 'better-sqlite3'; +import { ExternalNodeEnrichment, UpsertExternalNodeEnrichmentInput } from '../../db/types'; +import { IExternalNodeEnrichmentStore } from '../interfaces/IExternalNodeEnrichmentStore'; + +export class ExternalNodeEnrichmentStore implements IExternalNodeEnrichmentStore { + constructor(private db: Database) {} + + findByCanonicalName(name: string): ExternalNodeEnrichment | undefined { + return this.db + .prepare('SELECT * FROM external_node_enrichment WHERE canonical_name = ?') + .get(name) as ExternalNodeEnrichment | undefined; + } + + findAll(): ExternalNodeEnrichment[] { + return this.db + .prepare('SELECT * FROM external_node_enrichment ORDER BY canonical_name') + .all() as ExternalNodeEnrichment[]; + } + + upsert(input: UpsertExternalNodeEnrichmentInput): ExternalNodeEnrichment { + const id = randomUUID(); + + this.db + .prepare(` + INSERT INTO external_node_enrichment ( + id, canonical_name, display_name, description, impact, contact, service_type, updated_by + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(canonical_name) DO UPDATE SET + display_name = excluded.display_name, + description = excluded.description, + impact = excluded.impact, + contact = excluded.contact, + service_type = excluded.service_type, + updated_at = datetime('now'), + updated_by = excluded.updated_by + `) + .run( + id, + input.canonical_name, + input.display_name ?? null, + input.description ?? null, + input.impact ?? null, + input.contact ?? null, + input.service_type ?? null, + input.updated_by ?? null + ); + + return this.findByCanonicalName(input.canonical_name)!; + } + + delete(id: string): boolean { + const result = this.db + .prepare('DELETE FROM external_node_enrichment WHERE id = ?') + .run(id); + return result.changes > 0; + } +} diff --git a/server/src/stores/impl/LatencyHistoryStore.test.ts b/server/src/stores/impl/LatencyHistoryStore.test.ts index 83d088e..a2bb302 100644 --- a/server/src/stores/impl/LatencyHistoryStore.test.ts +++ b/server/src/stores/impl/LatencyHistoryStore.test.ts @@ -13,6 +13,13 @@ describe('LatencyHistoryStore', () => { id TEXT PRIMARY KEY, dependency_id TEXT NOT NULL, latency_ms INTEGER NOT NULL, + p50_ms REAL, + p95_ms REAL, + p99_ms REAL, + min_ms REAL, + max_ms REAL, + request_count INTEGER, + source TEXT NOT NULL DEFAULT 'poll', recorded_at TEXT NOT NULL ) `); @@ -291,4 +298,85 @@ describe('LatencyHistoryStore', () => { expect(store.getHistory(testDependencyId)).toHaveLength(0); }); }); + + describe('recordWithPercentiles', () => { + let fullDb: Database.Database; + let fullStore: LatencyHistoryStore; + + beforeEach(() => { + fullDb = new Database(':memory:'); + fullDb.exec(` + CREATE TABLE dependency_latency_history ( + id TEXT PRIMARY KEY, + dependency_id TEXT NOT NULL, + latency_ms INTEGER NOT NULL, + p50_ms REAL, + p95_ms REAL, + p99_ms REAL, + min_ms REAL, + max_ms REAL, + request_count INTEGER, + source TEXT NOT NULL DEFAULT 'poll', + recorded_at TEXT NOT NULL + ) + `); + fullStore = new LatencyHistoryStore(fullDb); + }); + + afterEach(() => { + fullDb.close(); + }); + + it('stores all percentile fields', () => { + const timestamp = new Date().toISOString(); + const result = fullStore.recordWithPercentiles( + testDependencyId, + 150, + { p50: 100, p95: 200, p99: 300, min: 10, max: 500, requestCount: 1000 }, + timestamp, + 'otlp_histogram' + ); + + expect(result.id).toBeDefined(); + expect(result.dependency_id).toBe(testDependencyId); + expect(result.latency_ms).toBe(150); + + // Verify in DB directly + const row = fullDb.prepare('SELECT * FROM dependency_latency_history WHERE id = ?').get(result.id) as Record; + expect(row.p50_ms).toBe(100); + expect(row.p95_ms).toBe(200); + expect(row.p99_ms).toBe(300); + expect(row.min_ms).toBe(10); + expect(row.max_ms).toBe(500); + expect(row.request_count).toBe(1000); + expect(row.source).toBe('otlp_histogram'); + }); + + it('handles partial percentile data', () => { + const timestamp = new Date().toISOString(); + fullStore.recordWithPercentiles( + testDependencyId, + 100, + { p50: 80, p95: 120 }, + timestamp, + 'otlp_trace' + ); + + const row = fullDb.prepare('SELECT * FROM dependency_latency_history').get() as Record; + expect(row.p50_ms).toBe(80); + expect(row.p95_ms).toBe(120); + expect(row.p99_ms).toBeNull(); + expect(row.min_ms).toBeNull(); + expect(row.source).toBe('otlp_trace'); + }); + + it('existing record() still works (backward compat)', () => { + const timestamp = new Date().toISOString(); + const result = fullStore.record(testDependencyId, 50, timestamp); + + expect(result.latency_ms).toBe(50); + const row = fullDb.prepare('SELECT source FROM dependency_latency_history WHERE id = ?').get(result.id) as { source: string }; + expect(row.source).toBe('poll'); + }); + }); }); diff --git a/server/src/stores/impl/LatencyHistoryStore.ts b/server/src/stores/impl/LatencyHistoryStore.ts index 6befb86..8fb54d7 100644 --- a/server/src/stores/impl/LatencyHistoryStore.ts +++ b/server/src/stores/impl/LatencyHistoryStore.ts @@ -6,6 +6,7 @@ import { LatencyDataPoint, LatencyBucket, LatencyRange, + PercentileInput, } from '../interfaces/ILatencyHistoryStore'; import { LatencyStats } from '../types'; @@ -69,6 +70,42 @@ export class LatencyHistoryStore implements ILatencyHistoryStore { .get(id) as DependencyLatencyHistory; } + recordWithPercentiles( + dependencyId: string, + latencyMs: number, + percentiles: PercentileInput, + timestamp: string, + source: string + ): DependencyLatencyHistory { + const id = randomUUID(); + + this.db + .prepare(` + INSERT INTO dependency_latency_history ( + id, dependency_id, latency_ms, + p50_ms, p95_ms, p99_ms, min_ms, max_ms, request_count, source, + recorded_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `) + .run( + id, + dependencyId, + latencyMs, + percentiles.p50 ?? null, + percentiles.p95 ?? null, + percentiles.p99 ?? null, + percentiles.min ?? null, + percentiles.max ?? null, + percentiles.requestCount ?? null, + source, + timestamp + ); + + return this.db + .prepare('SELECT * FROM dependency_latency_history WHERE id = ?') + .get(id) as DependencyLatencyHistory; + } + getStats24h(dependencyId: string): LatencyStats { const row = this.db .prepare(` @@ -155,7 +192,10 @@ export class LatencyHistoryStore implements ILatencyHistoryStore { MIN(latency_ms) as min, ROUND(AVG(latency_ms)) as avg, MAX(latency_ms) as max, - COUNT(*) as count + COUNT(*) as count, + ROUND(AVG(p50_ms)) as avg_p50, + ROUND(AVG(p95_ms)) as avg_p95, + ROUND(AVG(p99_ms)) as avg_p99 FROM dependency_latency_history WHERE dependency_id = ? AND recorded_at >= datetime('now', '${config.offset}') @@ -181,7 +221,10 @@ export class LatencyHistoryStore implements ILatencyHistoryStore { MIN(latency_ms) as min, ROUND(AVG(latency_ms)) as avg, MAX(latency_ms) as max, - COUNT(*) as count + COUNT(*) as count, + ROUND(AVG(p50_ms)) as avg_p50, + ROUND(AVG(p95_ms)) as avg_p95, + ROUND(AVG(p99_ms)) as avg_p99 FROM dependency_latency_history WHERE dependency_id IN (${placeholders}) AND recorded_at >= datetime('now', '${config.offset}') diff --git a/server/src/stores/impl/ServiceStore.test.ts b/server/src/stores/impl/ServiceStore.test.ts index 1280378..992dfbf 100644 --- a/server/src/stores/impl/ServiceStore.test.ts +++ b/server/src/stores/impl/ServiceStore.test.ts @@ -30,6 +30,7 @@ describe('ServiceStore', () => { is_active INTEGER NOT NULL DEFAULT 1, is_external INTEGER NOT NULL DEFAULT 0, description TEXT, + health_endpoint_format TEXT NOT NULL DEFAULT 'default', last_poll_success INTEGER, last_poll_error TEXT, poll_warnings TEXT, diff --git a/server/src/stores/impl/ServiceStore.ts b/server/src/stores/impl/ServiceStore.ts index 824e2a3..2758110 100644 --- a/server/src/stores/impl/ServiceStore.ts +++ b/server/src/stores/impl/ServiceStore.ts @@ -138,8 +138,8 @@ export class ServiceStore implements IServiceStore { this.db .prepare(` - INSERT INTO services (id, name, team_id, health_endpoint, metrics_endpoint, schema_config, poll_interval_ms, is_active, is_external, description, created_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?, ?, ?) + INSERT INTO services (id, name, team_id, health_endpoint, metrics_endpoint, schema_config, poll_interval_ms, is_active, is_external, description, health_endpoint_format, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?, ?, ?, ?) `) .run( id, @@ -151,6 +151,7 @@ export class ServiceStore implements IServiceStore { input.poll_interval_ms ?? 30000, isExternal, input.description ?? null, + input.health_endpoint_format ?? 'default', now, now ); @@ -200,6 +201,10 @@ export class ServiceStore implements IServiceStore { updates.push('description = ?'); params.push(input.description); } + if (input.health_endpoint_format !== undefined) { + updates.push('health_endpoint_format = ?'); + params.push(input.health_endpoint_format); + } if (updates.length === 0) { return existing; diff --git a/server/src/stores/impl/SpanStore.test.ts b/server/src/stores/impl/SpanStore.test.ts new file mode 100644 index 0000000..0527542 --- /dev/null +++ b/server/src/stores/impl/SpanStore.test.ts @@ -0,0 +1,155 @@ +import Database from 'better-sqlite3'; +import { runMigrations } from '../../db/migrate'; +import { SpanStore } from './SpanStore'; +import { CreateSpanInput } from '../../db/types'; + +describe('SpanStore', () => { + let db: Database.Database; + let store: SpanStore; + + beforeEach(() => { + db = new Database(':memory:'); + db.pragma('foreign_keys = ON'); + runMigrations(db); + db.prepare("INSERT INTO teams (id, name) VALUES ('team-1', 'Test Team')").run(); + store = new SpanStore(db); + }); + + afterEach(() => { + db.close(); + }); + + const makeSpan = (overrides: Partial = {}): CreateSpanInput => ({ + trace_id: 'trace-1', + span_id: 'span-1', + service_name: 'my-service', + team_id: 'team-1', + name: 'GET /api/users', + kind: 3, // CLIENT + start_time: '2024-01-01T00:00:00Z', + end_time: '2024-01-01T00:00:00.100Z', + duration_ms: 100, + ...overrides, + }); + + describe('bulkInsert', () => { + it('returns 0 for empty array', () => { + expect(store.bulkInsert([])).toBe(0); + }); + + it('inserts multiple spans', () => { + const spans = [ + makeSpan({ span_id: 'span-1' }), + makeSpan({ span_id: 'span-2', name: 'POST /api/orders' }), + makeSpan({ span_id: 'span-3', name: 'GET /api/items', kind: 2 }), + ]; + + const count = store.bulkInsert(spans); + expect(count).toBe(3); + + const all = db.prepare('SELECT * FROM spans').all(); + expect(all).toHaveLength(3); + }); + + it('generates unique IDs for each span', () => { + store.bulkInsert([makeSpan({ span_id: 'a' }), makeSpan({ span_id: 'b' })]); + + const rows = db.prepare('SELECT id FROM spans').all() as { id: string }[]; + expect(rows[0].id).not.toBe(rows[1].id); + }); + + it('stores attributes as provided', () => { + const attrs = JSON.stringify({ 'http.method': 'GET' }); + store.bulkInsert([makeSpan({ attributes: attrs })]); + + const row = db.prepare('SELECT attributes FROM spans').get() as { attributes: string }; + expect(row.attributes).toBe(attrs); + }); + }); + + describe('findByTraceId', () => { + it('returns spans for a given trace ordered by start_time', () => { + store.bulkInsert([ + makeSpan({ span_id: 'late', start_time: '2024-01-01T00:00:02Z' }), + makeSpan({ span_id: 'early', start_time: '2024-01-01T00:00:00Z' }), + makeSpan({ span_id: 'mid', start_time: '2024-01-01T00:00:01Z' }), + ]); + + const spans = store.findByTraceId('trace-1'); + expect(spans).toHaveLength(3); + expect(spans[0].span_id).toBe('early'); + expect(spans[1].span_id).toBe('mid'); + expect(spans[2].span_id).toBe('late'); + }); + + it('returns empty for non-existent trace', () => { + expect(store.findByTraceId('no-such-trace')).toHaveLength(0); + }); + + it('does not return spans from other traces', () => { + store.bulkInsert([ + makeSpan({ trace_id: 'trace-1', span_id: 's1' }), + makeSpan({ trace_id: 'trace-2', span_id: 's2' }), + ]); + + const spans = store.findByTraceId('trace-1'); + expect(spans).toHaveLength(1); + expect(spans[0].span_id).toBe('s1'); + }); + }); + + describe('findByServiceName', () => { + it('filters by service name', () => { + store.bulkInsert([ + makeSpan({ service_name: 'svc-a', span_id: 's1' }), + makeSpan({ service_name: 'svc-b', span_id: 's2' }), + ]); + + const spans = store.findByServiceName('svc-a'); + expect(spans).toHaveLength(1); + expect(spans[0].span_id).toBe('s1'); + }); + + it('filters by since timestamp', () => { + store.bulkInsert([ + makeSpan({ span_id: 'old', start_time: '2024-01-01T00:00:00Z' }), + makeSpan({ span_id: 'new', start_time: '2024-06-01T00:00:00Z' }), + ]); + + const spans = store.findByServiceName('my-service', { since: '2024-03-01T00:00:00Z' }); + expect(spans).toHaveLength(1); + expect(spans[0].span_id).toBe('new'); + }); + + it('respects limit', () => { + store.bulkInsert([ + makeSpan({ span_id: 's1' }), + makeSpan({ span_id: 's2' }), + makeSpan({ span_id: 's3' }), + ]); + + const spans = store.findByServiceName('my-service', { limit: 2 }); + expect(spans).toHaveLength(2); + }); + }); + + describe('deleteOlderThan', () => { + it('removes old spans and keeps recent ones', () => { + store.bulkInsert([ + makeSpan({ span_id: 'old' }), + makeSpan({ span_id: 'new' }), + ]); + + // Update created_at to simulate old vs new + db.prepare("UPDATE spans SET created_at = '2024-01-01T00:00:00Z' WHERE span_id = 'old'").run(); + db.prepare("UPDATE spans SET created_at = '2024-12-01T00:00:00Z' WHERE span_id = 'new'").run(); + + const deleted = store.deleteOlderThan('2024-06-01T00:00:00Z'); + expect(deleted).toBe(1); + + const remaining = db.prepare('SELECT span_id FROM spans').all() as { span_id: string }[]; + expect(remaining).toHaveLength(1); + expect(remaining[0].span_id).toBe('new'); + }); + }); +}); diff --git a/server/src/stores/impl/SpanStore.ts b/server/src/stores/impl/SpanStore.ts new file mode 100644 index 0000000..ac8c075 --- /dev/null +++ b/server/src/stores/impl/SpanStore.ts @@ -0,0 +1,80 @@ +import { randomUUID } from 'crypto'; +import { Database } from 'better-sqlite3'; +import { Span, CreateSpanInput } from '../../db/types'; +import { ISpanStore } from '../interfaces/ISpanStore'; + +export class SpanStore implements ISpanStore { + constructor(private db: Database) {} + + bulkInsert(spans: CreateSpanInput[]): number { + if (spans.length === 0) return 0; + + const stmt = this.db.prepare(` + INSERT INTO spans ( + id, trace_id, span_id, parent_span_id, service_name, team_id, + name, kind, start_time, end_time, duration_ms, + status_code, status_message, attributes, resource_attributes + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `); + + const insertAll = this.db.transaction((items: CreateSpanInput[]) => { + let count = 0; + for (const span of items) { + stmt.run( + randomUUID(), + span.trace_id, + span.span_id, + span.parent_span_id ?? null, + span.service_name, + span.team_id, + span.name, + span.kind ?? 0, + span.start_time, + span.end_time, + span.duration_ms, + span.status_code ?? 0, + span.status_message ?? null, + span.attributes ?? null, + span.resource_attributes ?? null + ); + count++; + } + return count; + }); + + return insertAll(spans); + } + + findByTraceId(traceId: string): Span[] { + return this.db + .prepare('SELECT * FROM spans WHERE trace_id = ? ORDER BY start_time ASC') + .all(traceId) as Span[]; + } + + findByServiceName( + serviceName: string, + options?: { since?: string; limit?: number } + ): Span[] { + const conditions = ['service_name = ?']; + const params: unknown[] = [serviceName]; + + if (options?.since) { + conditions.push('start_time >= ?'); + params.push(options.since); + } + + const limit = options?.limit ?? 1000; + return this.db + .prepare( + `SELECT * FROM spans WHERE ${conditions.join(' AND ')} ORDER BY start_time DESC LIMIT ?` + ) + .all(...params, limit) as Span[]; + } + + deleteOlderThan(timestamp: string): number { + const result = this.db + .prepare('DELETE FROM spans WHERE created_at < ?') + .run(timestamp); + return result.changes; + } +} diff --git a/server/src/stores/impl/TeamApiKeyStore.test.ts b/server/src/stores/impl/TeamApiKeyStore.test.ts new file mode 100644 index 0000000..be83629 --- /dev/null +++ b/server/src/stores/impl/TeamApiKeyStore.test.ts @@ -0,0 +1,259 @@ +import Database from 'better-sqlite3'; +import { createHash } from 'crypto'; +import { TeamApiKeyStore } from './TeamApiKeyStore'; + +describe('TeamApiKeyStore', () => { + let db: Database.Database; + let store: TeamApiKeyStore; + + beforeEach(() => { + db = new Database(':memory:'); + db.exec(` + CREATE TABLE team_api_keys ( + id TEXT PRIMARY KEY, + team_id TEXT NOT NULL, + name TEXT NOT NULL, + key_hash TEXT NOT NULL, + key_prefix TEXT NOT NULL, + rate_limit_rpm INTEGER, + rate_limit_admin_locked INTEGER NOT NULL DEFAULT 0, + last_used_at TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + created_by TEXT + ); + CREATE UNIQUE INDEX idx_team_api_keys_key_hash ON team_api_keys(key_hash); + `); + store = new TeamApiKeyStore(db); + }); + + afterEach(() => { + db.close(); + }); + + describe('create', () => { + it('should return a record with rawKey', () => { + const result = store.create({ + team_id: 'team-1', + name: 'Test Key', + created_by: 'user-1', + }); + + expect(result.id).toBeDefined(); + expect(result.rawKey).toBeDefined(); + expect(result.team_id).toBe('team-1'); + expect(result.name).toBe('Test Key'); + expect(result.created_by).toBe('user-1'); + expect(result.last_used_at).toBeNull(); + }); + + it('should generate key with dps_ prefix and 32 hex chars', () => { + const result = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + expect(result.rawKey).toMatch(/^dps_[0-9a-f]{32}$/); + }); + + it('should store SHA-256 hash of the raw key', () => { + const result = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + const expectedHash = createHash('sha256') + .update(result.rawKey) + .digest('hex'); + expect(result.key_hash).toBe(expectedHash); + }); + + it('should store first 8 chars as key_prefix', () => { + const result = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + expect(result.key_prefix).toBe(result.rawKey.slice(0, 8)); + }); + + it('should set created_by to null when not provided', () => { + const result = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + expect(result.created_by).toBeNull(); + }); + }); + + describe('findByTeamId', () => { + it('should return keys for the specified team', () => { + store.create({ team_id: 'team-1', name: 'Key A' }); + store.create({ team_id: 'team-1', name: 'Key B' }); + store.create({ team_id: 'team-2', name: 'Key C' }); + + const keys = store.findByTeamId('team-1'); + expect(keys).toHaveLength(2); + expect(keys.every((k) => k.team_id === 'team-1')).toBe(true); + }); + + it('should return empty array when team has no keys', () => { + const keys = store.findByTeamId('team-nonexistent'); + expect(keys).toEqual([]); + }); + + it('should order by created_at descending', () => { + const first = store.create({ team_id: 'team-1', name: 'First' }); + // Manually set an older created_at so ordering is deterministic + db.prepare( + `UPDATE team_api_keys SET created_at = '2024-01-01T00:00:00' WHERE id = ?`, + ).run(first.id); + + store.create({ team_id: 'team-1', name: 'Second' }); + + const keys = store.findByTeamId('team-1'); + // Most recent first + expect(keys[0].name).toBe('Second'); + expect(keys[1].name).toBe('First'); + }); + }); + + describe('findByKeyHash', () => { + it('should find key by its hash', () => { + const created = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + const hash = createHash('sha256') + .update(created.rawKey) + .digest('hex'); + const found = store.findByKeyHash(hash); + + expect(found).toBeDefined(); + expect(found!.id).toBe(created.id); + }); + + it('should return undefined for non-existent hash', () => { + const found = store.findByKeyHash('nonexistent-hash'); + expect(found).toBeUndefined(); + }); + }); + + describe('delete', () => { + it('should remove the key and return true', () => { + const created = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + const result = store.delete(created.id); + expect(result).toBe(true); + + const keys = store.findByTeamId('team-1'); + expect(keys).toHaveLength(0); + }); + + it('should return false when key does not exist', () => { + const result = store.delete('nonexistent-id'); + expect(result).toBe(false); + }); + }); + + describe('updateLastUsed', () => { + it('should update the last_used_at timestamp', () => { + const created = store.create({ + team_id: 'team-1', + name: 'Test Key', + }); + + expect(created.last_used_at).toBeNull(); + + store.updateLastUsed(created.id); + + const hash = createHash('sha256') + .update(created.rawKey) + .digest('hex'); + const updated = store.findByKeyHash(hash); + + expect(updated!.last_used_at).not.toBeNull(); + }); + }); + + describe('updateRateLimit (DPS-100m)', () => { + it('should persist a custom rate_limit_rpm value', () => { + const created = store.create({ team_id: 'team-1', name: 'Test Key' }); + + const updated = store.updateRateLimit(created.id, 600); + + expect(updated.rate_limit_rpm).toBe(600); + + const found = store.findById(created.id); + expect(found!.rate_limit_rpm).toBe(600); + }); + + it('should reset rate_limit_rpm to null', () => { + const created = store.create({ team_id: 'team-1', name: 'Test Key' }); + + // Set to custom first + store.updateRateLimit(created.id, 600); + // Reset to null + const updated = store.updateRateLimit(created.id, null); + + expect(updated.rate_limit_rpm).toBeNull(); + + const found = store.findById(created.id); + expect(found!.rate_limit_rpm).toBeNull(); + }); + + it('should throw when key does not exist', () => { + expect(() => store.updateRateLimit('nonexistent-id', 600)).toThrow('API key not found'); + }); + }); + + describe('setAdminLock (DPS-100m)', () => { + it('should set rate_limit_admin_locked to 1 when locked', () => { + const created = store.create({ team_id: 'team-1', name: 'Test Key' }); + + const updated = store.setAdminLock(created.id, true); + + expect(updated.rate_limit_admin_locked).toBe(1); + }); + + it('should set both lock and rate limit atomically', () => { + const created = store.create({ team_id: 'team-1', name: 'Test Key' }); + + const updated = store.setAdminLock(created.id, true, 1200); + + expect(updated.rate_limit_admin_locked).toBe(1); + expect(updated.rate_limit_rpm).toBe(1200); + }); + + it('should clear rate_limit_admin_locked to 0 when unlocked', () => { + const created = store.create({ team_id: 'team-1', name: 'Test Key' }); + + // Lock first + store.setAdminLock(created.id, true, 1200); + // Unlock + const updated = store.setAdminLock(created.id, false); + + expect(updated.rate_limit_admin_locked).toBe(0); + // rate_limit_rpm should be preserved since no rateLimit was passed + expect(updated.rate_limit_rpm).toBe(1200); + }); + + it('should not change rate_limit_rpm when rateLimit is not provided', () => { + const created = store.create({ team_id: 'team-1', name: 'Test Key' }); + + store.updateRateLimit(created.id, 500); + const updated = store.setAdminLock(created.id, true); + + expect(updated.rate_limit_admin_locked).toBe(1); + expect(updated.rate_limit_rpm).toBe(500); + }); + + it('should throw when key does not exist', () => { + expect(() => store.setAdminLock('nonexistent-id', true)).toThrow('API key not found'); + }); + }); +}); diff --git a/server/src/stores/impl/TeamApiKeyStore.ts b/server/src/stores/impl/TeamApiKeyStore.ts new file mode 100644 index 0000000..55b4958 --- /dev/null +++ b/server/src/stores/impl/TeamApiKeyStore.ts @@ -0,0 +1,86 @@ +import { randomUUID, randomBytes, createHash } from 'crypto'; +import { Database } from 'better-sqlite3'; +import { TeamApiKey, CreateTeamApiKeyInput } from '../../db/types'; +import { ITeamApiKeyStore } from '../interfaces/ITeamApiKeyStore'; + +export class TeamApiKeyStore implements ITeamApiKeyStore { + constructor(private db: Database) {} + + findByTeamId(teamId: string): TeamApiKey[] { + return this.db + .prepare( + `SELECT * FROM team_api_keys WHERE team_id = ? ORDER BY created_at DESC`, + ) + .all(teamId) as TeamApiKey[]; + } + + findByKeyHash(hash: string): TeamApiKey | undefined { + return this.db + .prepare(`SELECT * FROM team_api_keys WHERE key_hash = ?`) + .get(hash) as TeamApiKey | undefined; + } + + create(input: CreateTeamApiKeyInput): TeamApiKey & { rawKey: string } { + const id = randomUUID(); + const rawKey = `dps_${randomBytes(16).toString('hex')}`; + const keyHash = createHash('sha256').update(rawKey).digest('hex'); + const keyPrefix = rawKey.slice(0, 8); + + this.db + .prepare( + `INSERT INTO team_api_keys (id, team_id, name, key_hash, key_prefix, created_by) + VALUES (?, ?, ?, ?, ?, ?)`, + ) + .run(id, input.team_id, input.name, keyHash, keyPrefix, input.created_by ?? null); + + const record = this.db + .prepare(`SELECT * FROM team_api_keys WHERE id = ?`) + .get(id) as TeamApiKey; + + return { ...record, rawKey }; + } + + delete(id: string): boolean { + const result = this.db + .prepare(`DELETE FROM team_api_keys WHERE id = ?`) + .run(id); + return result.changes > 0; + } + + findById(id: string): TeamApiKey | undefined { + return this.db + .prepare(`SELECT * FROM team_api_keys WHERE id = ?`) + .get(id) as TeamApiKey | undefined; + } + + updateLastUsed(id: string): void { + this.db + .prepare( + `UPDATE team_api_keys SET last_used_at = datetime('now') WHERE id = ?`, + ) + .run(id); + } + + updateRateLimit(id: string, rateLimit: number | null): TeamApiKey { + this.db + .prepare(`UPDATE team_api_keys SET rate_limit_rpm = ? WHERE id = ?`) + .run(rateLimit, id); + const row = this.findById(id); + if (!row) throw new Error(`API key not found: ${id}`); + return row; + } + + setAdminLock(id: string, locked: boolean, rateLimit?: number | null): TeamApiKey { + this.db + .prepare( + `UPDATE team_api_keys + SET rate_limit_admin_locked = ?, + rate_limit_rpm = COALESCE(?, rate_limit_rpm) + WHERE id = ?`, + ) + .run(locked ? 1 : 0, rateLimit ?? null, id); + const row = this.findById(id); + if (!row) throw new Error(`API key not found: ${id}`); + return row; + } +} diff --git a/server/src/stores/index.ts b/server/src/stores/index.ts index 4a11a0d..95fc740 100644 --- a/server/src/stores/index.ts +++ b/server/src/stores/index.ts @@ -22,6 +22,11 @@ import type { IManifestConfigStore } from './interfaces/IManifestConfigStore'; import type { IManifestSyncHistoryStore } from './interfaces/IManifestSyncHistoryStore'; import type { IDriftFlagStore } from './interfaces/IDriftFlagStore'; import type { IAlertMuteStore } from './interfaces/IAlertMuteStore'; +import type { ITeamApiKeyStore } from './interfaces/ITeamApiKeyStore'; +import type { IApiKeyUsageStore } from './interfaces/IApiKeyUsageStore'; +import type { ISpanStore } from './interfaces/ISpanStore'; +import type { IAppSettingsStore } from './interfaces/IAppSettingsStore'; +import type { IExternalNodeEnrichmentStore } from './interfaces/IExternalNodeEnrichmentStore'; // Import implementations import { ServiceStore } from './impl/ServiceStore'; @@ -44,6 +49,11 @@ import { ManifestConfigStore } from './impl/ManifestConfigStore'; import { ManifestSyncHistoryStore } from './impl/ManifestSyncHistoryStore'; import { DriftFlagStore } from './impl/DriftFlagStore'; import { AlertMuteStore } from './impl/AlertMuteStore'; +import { TeamApiKeyStore } from './impl/TeamApiKeyStore'; +import { ApiKeyUsageStore } from './impl/ApiKeyUsageStore'; +import { SpanStore } from './impl/SpanStore'; +import { AppSettingsStore } from './impl/AppSettingsStore'; +import { ExternalNodeEnrichmentStore } from './impl/ExternalNodeEnrichmentStore'; /** * Central registry providing access to all stores. @@ -72,6 +82,11 @@ export class StoreRegistry { public readonly manifestSyncHistory: IManifestSyncHistoryStore; public readonly driftFlags: IDriftFlagStore; public readonly alertMutes: IAlertMuteStore; + public readonly teamApiKeys: ITeamApiKeyStore; + public readonly apiKeyUsage: IApiKeyUsageStore; + public readonly spans: ISpanStore; + public readonly appSettings: IAppSettingsStore; + public readonly externalNodeEnrichment: IExternalNodeEnrichmentStore; private constructor(database: Database) { this.services = new ServiceStore(database); @@ -94,6 +109,11 @@ export class StoreRegistry { this.manifestSyncHistory = new ManifestSyncHistoryStore(database); this.driftFlags = new DriftFlagStore(database); this.alertMutes = new AlertMuteStore(database); + this.teamApiKeys = new TeamApiKeyStore(database); + this.apiKeyUsage = new ApiKeyUsageStore(database); + this.spans = new SpanStore(database); + this.appSettings = new AppSettingsStore(database); + this.externalNodeEnrichment = new ExternalNodeEnrichmentStore(database); } /** diff --git a/server/src/stores/interfaces/IApiKeyUsageStore.ts b/server/src/stores/interfaces/IApiKeyUsageStore.ts new file mode 100644 index 0000000..143ee11 --- /dev/null +++ b/server/src/stores/interfaces/IApiKeyUsageStore.ts @@ -0,0 +1,20 @@ +import { ApiKeyUsageBucket } from '../../db/types'; + +export interface BulkUpsertEntry { + api_key_id: string; + bucket_start: string; + granularity: 'minute' | 'hour'; + push_count: number; + rejected_count: number; +} + +export interface IApiKeyUsageStore { + bulkUpsert(entries: BulkUpsertEntry[]): void; + getBuckets(apiKeyId: string, granularity: 'minute' | 'hour', from: string, to: string): ApiKeyUsageBucket[]; + getBucketsByTeam(teamId: string, granularity: 'minute' | 'hour', from: string, to: string): (ApiKeyUsageBucket & { key_name: string; key_prefix: string })[]; + getAllBuckets(granularity: 'minute' | 'hour', from: string, to: string): (ApiKeyUsageBucket & { team_id: string; key_name: string })[]; + getSummaryForKeys(apiKeyIds: string[], from: string, to: string): Map; + pruneMinuteBuckets(olderThan: string): number; + pruneHourBuckets(olderThan: string): number; + pruneOrphanedBuckets(olderThan: string): number; +} diff --git a/server/src/stores/interfaces/IAppSettingsStore.ts b/server/src/stores/interfaces/IAppSettingsStore.ts new file mode 100644 index 0000000..05b9434 --- /dev/null +++ b/server/src/stores/interfaces/IAppSettingsStore.ts @@ -0,0 +1,4 @@ +export interface IAppSettingsStore { + get(key: string): string | undefined; + set(key: string, value: string, updatedBy?: string): void; +} diff --git a/server/src/stores/interfaces/IAssociationStore.ts b/server/src/stores/interfaces/IAssociationStore.ts index c5f5117..d1c439c 100644 --- a/server/src/stores/interfaces/IAssociationStore.ts +++ b/server/src/stores/interfaces/IAssociationStore.ts @@ -20,11 +20,23 @@ export interface IAssociationStore { */ existsForDependencyAndService(dependencyId: string, linkedServiceId: string): boolean; + // Auto-suggestion queries + findAutoSuggested(dependencyId: string): DependencyAssociation[]; + // Write operations create(input: AssociationCreateInput): DependencyAssociation; delete(id: string): boolean; deleteByDependencyId(dependencyId: string): number; + // Auto-suggestion mutations + confirm(id: string): boolean; + dismiss(id: string): boolean; + + /** + * Delete dismissed auto-suggested associations older than the given timestamp. + */ + deleteOldDismissed(olderThan: string): number; + // Utility exists(id: string): boolean; count(options?: AssociationListOptions): number; diff --git a/server/src/stores/interfaces/IDependencyStore.ts b/server/src/stores/interfaces/IDependencyStore.ts index 96fc80b..45435cc 100644 --- a/server/src/stores/interfaces/IDependencyStore.ts +++ b/server/src/stores/interfaces/IDependencyStore.ts @@ -5,6 +5,7 @@ import { DependencyListOptions, DependencyUpsertInput, DependencyOverrideInput, + DependencyUserEnrichmentInput, DependentReport, } from '../types'; @@ -65,9 +66,16 @@ export interface IDependencyStore { */ findDependentReports(serviceId: string): DependentReport[]; + /** + * Find dependencies for a service filtered by discovery source. + * Used for listing trace-discovered dependencies. + */ + findByDiscoverySource(serviceId: string, source: string): Dependency[]; + // Write operations upsert(input: DependencyUpsertInput): UpsertResult; updateOverrides(id: string, overrides: DependencyOverrideInput): Dependency | undefined; + updateUserEnrichment(id: string, enrichment: DependencyUserEnrichmentInput): Dependency | undefined; delete(id: string): boolean; deleteByServiceId(serviceId: string): number; diff --git a/server/src/stores/interfaces/IExternalNodeEnrichmentStore.ts b/server/src/stores/interfaces/IExternalNodeEnrichmentStore.ts new file mode 100644 index 0000000..8c0dc94 --- /dev/null +++ b/server/src/stores/interfaces/IExternalNodeEnrichmentStore.ts @@ -0,0 +1,8 @@ +import { ExternalNodeEnrichment, UpsertExternalNodeEnrichmentInput } from '../../db/types'; + +export interface IExternalNodeEnrichmentStore { + findByCanonicalName(name: string): ExternalNodeEnrichment | undefined; + findAll(): ExternalNodeEnrichment[]; + upsert(input: UpsertExternalNodeEnrichmentInput): ExternalNodeEnrichment; + delete(id: string): boolean; +} diff --git a/server/src/stores/interfaces/ILatencyHistoryStore.ts b/server/src/stores/interfaces/ILatencyHistoryStore.ts index a4c430e..44df45d 100644 --- a/server/src/stores/interfaces/ILatencyHistoryStore.ts +++ b/server/src/stores/interfaces/ILatencyHistoryStore.ts @@ -18,6 +18,9 @@ export interface LatencyBucket { avg: number; max: number; count: number; + avg_p50?: number | null; + avg_p95?: number | null; + avg_p99?: number | null; } /** @@ -28,12 +31,32 @@ export type LatencyRange = '1h' | '6h' | '24h' | '7d' | '30d'; /** * Store interface for DependencyLatencyHistory entity operations */ +export interface PercentileInput { + p50?: number; + p95?: number; + p99?: number; + min?: number; + max?: number; + requestCount?: number; +} + export interface ILatencyHistoryStore { /** * Record a new latency measurement */ record(dependencyId: string, latencyMs: number, timestamp: string): DependencyLatencyHistory; + /** + * Record a latency measurement with histogram-derived percentiles + */ + recordWithPercentiles( + dependencyId: string, + latencyMs: number, + percentiles: PercentileInput, + timestamp: string, + source: string + ): DependencyLatencyHistory; + /** * Get latency statistics for the last 24 hours */ diff --git a/server/src/stores/interfaces/ISpanStore.ts b/server/src/stores/interfaces/ISpanStore.ts new file mode 100644 index 0000000..4e23ed4 --- /dev/null +++ b/server/src/stores/interfaces/ISpanStore.ts @@ -0,0 +1,8 @@ +import { Span, CreateSpanInput } from '../../db/types'; + +export interface ISpanStore { + bulkInsert(spans: CreateSpanInput[]): number; + findByTraceId(traceId: string): Span[]; + findByServiceName(serviceName: string, options?: { since?: string; limit?: number }): Span[]; + deleteOlderThan(timestamp: string): number; +} diff --git a/server/src/stores/interfaces/ITeamApiKeyStore.ts b/server/src/stores/interfaces/ITeamApiKeyStore.ts new file mode 100644 index 0000000..1e66ca9 --- /dev/null +++ b/server/src/stores/interfaces/ITeamApiKeyStore.ts @@ -0,0 +1,12 @@ +import { TeamApiKey, CreateTeamApiKeyInput } from '../../db/types'; + +export interface ITeamApiKeyStore { + findByTeamId(teamId: string): TeamApiKey[]; + findByKeyHash(hash: string): TeamApiKey | undefined; + findById(id: string): TeamApiKey | undefined; + create(input: CreateTeamApiKeyInput): TeamApiKey & { rawKey: string }; + delete(id: string): boolean; + updateLastUsed(id: string): void; + updateRateLimit(id: string, rateLimit: number | null): TeamApiKey; + setAdminLock(id: string, locked: boolean, rateLimit?: number | null): TeamApiKey; +} diff --git a/server/src/stores/interfaces/index.ts b/server/src/stores/interfaces/index.ts index 6d0b156..33acb72 100644 --- a/server/src/stores/interfaces/index.ts +++ b/server/src/stores/interfaces/index.ts @@ -3,7 +3,7 @@ export type { ITeamStore, TeamMemberWithUser } from './ITeamStore'; export type { IUserStore } from './IUserStore'; export type { IDependencyStore, ExistingDependency, UpsertResult } from './IDependencyStore'; export type { IAssociationStore } from './IAssociationStore'; -export type { ILatencyHistoryStore, LatencyDataPoint } from './ILatencyHistoryStore'; +export type { ILatencyHistoryStore, LatencyDataPoint, PercentileInput } from './ILatencyHistoryStore'; export type { IErrorHistoryStore } from './IErrorHistoryStore'; export type { IDependencyAliasStore } from './IDependencyAliasStore'; export type { IAuditLogStore, AuditLogListOptions } from './IAuditLogStore'; @@ -18,3 +18,8 @@ export type { IManifestConfigStore, ManifestSyncResultInput } from './IManifestC export type { IManifestSyncHistoryStore, ManifestSyncHistoryCreateInput } from './IManifestSyncHistoryStore'; export type { IDriftFlagStore, DriftFlagListOptions } from './IDriftFlagStore'; export type { IAlertMuteStore } from './IAlertMuteStore'; +export type { ITeamApiKeyStore } from './ITeamApiKeyStore'; +export type { IApiKeyUsageStore, BulkUpsertEntry } from './IApiKeyUsageStore'; +export type { ISpanStore } from './ISpanStore'; +export type { IAppSettingsStore } from './IAppSettingsStore'; +export type { IExternalNodeEnrichmentStore } from './IExternalNodeEnrichmentStore'; diff --git a/server/src/stores/types.ts b/server/src/stores/types.ts index 461a1d1..a81b72a 100644 --- a/server/src/stores/types.ts +++ b/server/src/stores/types.ts @@ -7,6 +7,7 @@ import { AssociationType, TeamMemberRole, HealthState, + HealthEndpointFormat, } from '../db/types'; // Database context for dependency injection @@ -74,8 +75,10 @@ export interface DependencyWithResolvedOverrides extends Dependency { export interface DependencyWithTarget extends Dependency { service_name: string; target_service_id: string | null; + association_id: string | null; association_type: AssociationType | null; avg_latency_24h: number | null; + is_auto_suggested: number | null; } /** @@ -151,6 +154,7 @@ export interface ServiceCreateInput { poll_interval_ms?: number; is_external?: boolean; description?: string | null; + health_endpoint_format?: HealthEndpointFormat; } export interface ServiceUpdateInput { @@ -162,6 +166,7 @@ export interface ServiceUpdateInput { poll_interval_ms?: number; is_active?: boolean; description?: string | null; + health_endpoint_format?: HealthEndpointFormat; } export interface TeamCreateInput { @@ -210,6 +215,7 @@ export interface DependencyUpsertInput { error_message?: string | null; skipped?: boolean; last_checked: string; + discovery_source?: 'manual' | 'otlp_metric' | 'otlp_trace'; } export interface DependencyOverrideInput { @@ -217,6 +223,12 @@ export interface DependencyOverrideInput { impact_override?: string | null; } +export interface DependencyUserEnrichmentInput { + displayName?: string | null; + description?: string | null; + impact?: string | null; +} + /** * Service-level poll history entry */ @@ -231,4 +243,5 @@ export interface AssociationCreateInput { dependency_id: string; linked_service_id: string; association_type: AssociationType; + is_auto_suggested?: boolean; } diff --git a/server/src/utils/dependencyOverrideResolver.test.ts b/server/src/utils/dependencyOverrideResolver.test.ts index c66bc52..6de1793 100644 --- a/server/src/utils/dependencyOverrideResolver.test.ts +++ b/server/src/utils/dependencyOverrideResolver.test.ts @@ -20,6 +20,10 @@ function makeDependency(overrides: Partial = {}): Dependency { check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: '2026-01-01T00:00:00.000Z', last_status_change: null, diff --git a/server/src/utils/duration.ts b/server/src/utils/duration.ts index dff869c..0c6ac91 100644 --- a/server/src/utils/duration.ts +++ b/server/src/utils/duration.ts @@ -32,6 +32,6 @@ export function parseDuration(duration: string): Date { throw new Error('Duration value must be positive'); } - const ms = value * UNIT_MS[unit]; + const ms = value * UNIT_MS[unit]; // eslint-disable-line security/detect-object-injection return new Date(Date.now() + ms); } diff --git a/server/src/utils/histogramPercentiles.test.ts b/server/src/utils/histogramPercentiles.test.ts new file mode 100644 index 0000000..46e6a6a --- /dev/null +++ b/server/src/utils/histogramPercentiles.test.ts @@ -0,0 +1,147 @@ +import { computePercentiles, HistogramBuckets } from './histogramPercentiles'; + +describe('computePercentiles', () => { + it('returns zeros for an empty histogram (count=0)', () => { + const buckets: HistogramBuckets = { + explicitBounds: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1], + bucketCounts: [0, 0, 0, 0, 0, 0, 0, 0, 0], + count: 0, + }; + + const result = computePercentiles(buckets); + + expect(result.p50).toBe(0); + expect(result.p95).toBe(0); + expect(result.p99).toBe(0); + expect(result.min).toBe(0); + expect(result.max).toBe(0); + expect(result.count).toBe(0); + expect(result.avgMs).toBe(0); + }); + + it('computes percentiles for a uniform distribution', () => { + // 100 requests uniformly spread across 10 buckets (10 per bucket) + // Bounds: [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] ms + const buckets: HistogramBuckets = { + explicitBounds: [10, 20, 30, 40, 50, 60, 70, 80, 90, 100], + bucketCounts: [10, 10, 10, 10, 10, 10, 10, 10, 10, 10], + count: 100, + sum: 50, // sum in same unit as bounds (ms here since multiplier=1) + }; + + const result = computePercentiles(buckets, 1); + + // p50 should be around 50ms (median of uniform 0-100) + expect(result.p50).toBeGreaterThanOrEqual(45); + expect(result.p50).toBeLessThanOrEqual(55); + + // p95 should be around 95ms + expect(result.p95).toBeGreaterThanOrEqual(90); + expect(result.p95).toBeLessThanOrEqual(100); + + // p99 should be around 99ms + expect(result.p99).toBeGreaterThanOrEqual(95); + expect(result.p99).toBeLessThanOrEqual(100); + + expect(result.count).toBe(100); + }); + + it('computes percentiles for a single-bucket histogram', () => { + // All 50 requests in the [0, 0.1) bucket → 0 to 100ms range + const buckets: HistogramBuckets = { + explicitBounds: [0.1], + bucketCounts: [50, 0], // 50 in first bucket, 0 in overflow + count: 50, + }; + + const result = computePercentiles(buckets); + + // All values within [0, 100ms], interpolation within that range + expect(result.p50).toBe(50); // 0.5 * 100ms + expect(result.p95).toBe(95); // 0.95 * 100ms + expect(result.p99).toBe(99); // 0.99 * 100ms + expect(result.count).toBe(50); + }); + + it('handles overflow bucket by capping at last explicit bound', () => { + // Most values in the overflow bucket (beyond last bound) + const buckets: HistogramBuckets = { + explicitBounds: [0.1, 0.5, 1], + bucketCounts: [0, 0, 0, 100], // all in overflow + count: 100, + }; + + const result = computePercentiles(buckets); + + // Overflow bucket interpolation is capped at last bound (1s = 1000ms) + // Since lower bound is also 1 (bounds[2]) and upper bound capped at 1 (bounds[2]), + // all values should equal 1000ms + expect(result.p50).toBe(1000); + expect(result.p95).toBe(1000); + expect(result.p99).toBe(1000); + }); + + it('applies seconds-to-ms conversion with default unitMultiplier', () => { + // OTel convention: bounds in seconds + const buckets: HistogramBuckets = { + explicitBounds: [0.005, 0.01, 0.025, 0.05, 0.1], + bucketCounts: [0, 0, 100, 0, 0, 0], // all in [0.01, 0.025) bucket + count: 100, + sum: 1.75, // average of 17.5ms in seconds + }; + + // Default multiplier = 1000 + const result = computePercentiles(buckets); + + // p50 should be in the [10ms, 25ms] range + expect(result.p50).toBeGreaterThanOrEqual(10); + expect(result.p50).toBeLessThanOrEqual(25); + + // avgMs should be sum/count * 1000 = 17.5 + expect(result.avgMs).toBe(17.5); + }); + + it('uses min/max from histogram data when provided', () => { + const buckets: HistogramBuckets = { + explicitBounds: [0.1, 0.5, 1], + bucketCounts: [10, 80, 10, 0], + count: 100, + min: 0.002, // 2ms + max: 0.95, // 950ms + }; + + const result = computePercentiles(buckets); + + expect(result.min).toBe(2); // 0.002 * 1000 + expect(result.max).toBe(950); // 0.95 * 1000 + }); + + it('computes count from bucketCounts when count field is absent', () => { + const buckets: HistogramBuckets = { + explicitBounds: [10, 20], + bucketCounts: [5, 10, 3], + // no count field + }; + + const result = computePercentiles(buckets, 1); + + expect(result.count).toBe(18); // 5 + 10 + 3 + }); + + it('handles skewed distribution correctly', () => { + // Most values in the first bucket (fast requests) + const buckets: HistogramBuckets = { + explicitBounds: [0.01, 0.05, 0.1, 0.5, 1], + bucketCounts: [950, 30, 10, 8, 2, 0], + count: 1000, + }; + + const result = computePercentiles(buckets); + + // p50 should be within first bucket [0, 10ms] + expect(result.p50).toBeLessThanOrEqual(10); + + // p99 should be in the higher buckets + expect(result.p99).toBeGreaterThan(10); + }); +}); diff --git a/server/src/utils/histogramPercentiles.ts b/server/src/utils/histogramPercentiles.ts new file mode 100644 index 0000000..8444941 --- /dev/null +++ b/server/src/utils/histogramPercentiles.ts @@ -0,0 +1,113 @@ +/** + * Input: explicit-boundary histogram bucket data from OTLP. + */ +export interface HistogramBuckets { + explicitBounds: number[]; + bucketCounts: number[]; + sum?: number; + count?: number; + min?: number; + max?: number; +} + +/** + * Output: computed percentiles and summary statistics. + */ +export interface PercentileResult { + p50: number; + p95: number; + p99: number; + min: number; + max: number; + count: number; + avgMs: number; +} + +/** + * Compute percentile approximations from explicit-boundary histogram buckets + * using linear interpolation within each bucket. + * + * @param buckets - Histogram bucket data + * @param unitMultiplier - Multiplier to convert bucket bounds to milliseconds. + * Defaults to 1000 (OTel convention: latency in seconds → ms). + * Pass 1 if bounds are already in ms. + */ +export function computePercentiles( + buckets: HistogramBuckets, + unitMultiplier: number = 1000, +): PercentileResult { + const { explicitBounds, bucketCounts, sum, count, min, max } = buckets; + + // Empty histogram — return zeros + const totalCount = count ?? bucketCounts.reduce((a, b) => a + b, 0); + if (totalCount === 0) { + return { p50: 0, p95: 0, p99: 0, min: 0, max: 0, count: 0, avgMs: 0 }; + } + + // Build cumulative counts + const cumulative: number[] = []; + let running = 0; + for (const c of bucketCounts) { + running += c; + cumulative.push(running); + } + + const p50 = interpolatePercentile(0.50, explicitBounds, cumulative, totalCount, unitMultiplier); + const p95 = interpolatePercentile(0.95, explicitBounds, cumulative, totalCount, unitMultiplier); + const p99 = interpolatePercentile(0.99, explicitBounds, cumulative, totalCount, unitMultiplier); + + const resolvedMin = min !== undefined ? min * unitMultiplier : p50; + const resolvedMax = max !== undefined ? max * unitMultiplier : p99; + const avgMs = sum !== undefined && totalCount > 0 + ? (sum / totalCount) * unitMultiplier + : p50; + + return { + p50: Math.round(p50 * 100) / 100, + p95: Math.round(p95 * 100) / 100, + p99: Math.round(p99 * 100) / 100, + min: Math.round(resolvedMin * 100) / 100, + max: Math.round(resolvedMax * 100) / 100, + count: totalCount, + avgMs: Math.round(avgMs * 100) / 100, + }; +} + +/** + * Linear interpolation within the bucket where the cumulative count + * crosses the target rank (count * percentile). + */ +function interpolatePercentile( + percentile: number, + bounds: number[], + cumulative: number[], + totalCount: number, + unitMultiplier: number, +): number { + const rank = totalCount * percentile; + + // Find the bucket where cumulative count crosses the rank + for (let i = 0; i < cumulative.length; i++) { + if (cumulative[i] >= rank) { // eslint-disable-line security/detect-object-injection + // Lower bound of the bucket + const lowerBound = i === 0 ? 0 : bounds[i - 1]; + // Upper bound — for the overflow bucket (last), cap at last explicit bound + const upperBound = i < bounds.length ? bounds[i] : bounds[bounds.length - 1]; // eslint-disable-line security/detect-object-injection + + const bucketCount = i === 0 ? cumulative[0] : cumulative[i] - cumulative[i - 1]; // eslint-disable-line security/detect-object-injection + if (bucketCount === 0) { + return lowerBound * unitMultiplier; + } + + // How far into this bucket the rank falls + const prevCumulative = i === 0 ? 0 : cumulative[i - 1]; + const fraction = (rank - prevCumulative) / bucketCount; + + const value = lowerBound + fraction * (upperBound - lowerBound); + return value * unitMultiplier; + } + } + + // Fallback: last bucket bound (shouldn't happen with valid data) + return (bounds[bounds.length - 1] ?? 0) * unitMultiplier; +} diff --git a/server/src/utils/serviceHealth.test.ts b/server/src/utils/serviceHealth.test.ts index 325482c..cd012ed 100644 --- a/server/src/utils/serviceHealth.test.ts +++ b/server/src/utils/serviceHealth.test.ts @@ -336,6 +336,10 @@ function createDependency( check_details: null, error: null, error_message: null, + discovery_source: 'manual', + user_display_name: null, + user_description: null, + user_impact: null, skipped: 0, last_checked: lastChecked || null, last_status_change: null, diff --git a/server/src/utils/validation.test.ts b/server/src/utils/validation.test.ts index 8a055d0..16cdc19 100644 --- a/server/src/utils/validation.test.ts +++ b/server/src/utils/validation.test.ts @@ -14,6 +14,7 @@ import { validateTeamMemberRoleUpdate, validateDependencyType, validateSchemaConfig, + validateMetricSchemaConfig, validateExternalServiceCreate, validateExternalServiceUpdate, MIN_POLL_INTERVAL_MS, @@ -284,6 +285,55 @@ describe('Service Validation', () => { expect(() => validateServiceCreate({ ...validInput, poll_interval_ms: 5000000 })) .toThrow(ValidationError); }); + + it('should accept valid health_endpoint_format values', () => { + for (const fmt of ['default', 'schema', 'prometheus']) { + const result = validateServiceCreate({ ...validInput, health_endpoint_format: fmt }); + expect(result.health_endpoint_format).toBe(fmt); + } + }); + + it('should not require health_endpoint_format (defaults handled elsewhere)', () => { + const result = validateServiceCreate(validInput); + expect(result.health_endpoint_format).toBeUndefined(); + }); + + it('should throw on invalid health_endpoint_format', () => { + expect(() => validateServiceCreate({ ...validInput, health_endpoint_format: 'invalid' })) + .toThrow(ValidationError); + expect(() => validateServiceCreate({ ...validInput, health_endpoint_format: 123 })) + .toThrow(ValidationError); + }); + + it('should allow OTLP format without health_endpoint', () => { + const result = validateServiceCreate({ + name: 'OTLP Service', + team_id: 'team-123', + health_endpoint_format: 'otlp', + }); + expect(result.health_endpoint_format).toBe('otlp'); + expect(result.health_endpoint).toBe(''); + expect(result.poll_interval_ms).toBe(0); + }); + + it('should reject health_endpoint for OTLP format', () => { + expect(() => validateServiceCreate({ + name: 'OTLP Service', + team_id: 'team-123', + health_endpoint: 'https://example.com/health', + health_endpoint_format: 'otlp', + })).toThrow(ValidationError); + }); + + it('should force poll_interval_ms to 0 for OTLP format', () => { + const result = validateServiceCreate({ + name: 'OTLP Service', + team_id: 'team-123', + health_endpoint_format: 'otlp', + poll_interval_ms: 60000, + }); + expect(result.poll_interval_ms).toBe(0); + }); }); describe('validateServiceUpdate', () => { @@ -356,6 +406,25 @@ describe('Service Validation', () => { expect(() => validateServiceUpdate({ is_active: 'false' })) .toThrow(ValidationError); }); + + it('should validate health_endpoint_format update', () => { + const result = validateServiceUpdate({ health_endpoint_format: 'prometheus' }); + expect(result?.health_endpoint_format).toBe('prometheus'); + }); + + it('should accept all valid format values in update', () => { + for (const fmt of ['default', 'schema', 'prometheus', 'otlp']) { + const result = validateServiceUpdate({ health_endpoint_format: fmt }); + expect(result?.health_endpoint_format).toBe(fmt); + } + }); + + it('should throw on invalid health_endpoint_format in update', () => { + expect(() => validateServiceUpdate({ health_endpoint_format: 'invalid' })) + .toThrow(ValidationError); + expect(() => validateServiceUpdate({ health_endpoint_format: 42 })) + .toThrow(ValidationError); + }); }); }); @@ -936,6 +1005,295 @@ describe('Schema Config Validation', () => { }); }); +describe('Metric Schema Config Validation', () => { + describe('validateMetricSchemaConfig', () => { + const validConfig = { + metrics: { http_status: 'state', is_up: 'healthy' }, + labels: { dep_name: 'name', dep_type: 'type' }, + }; + + it('should accept valid config with metrics and labels', () => { + const result = validateMetricSchemaConfig(validConfig); + const parsed = JSON.parse(result); + expect(parsed.metrics.http_status).toBe('state'); + expect(parsed.metrics.is_up).toBe('healthy'); + expect(parsed.labels.dep_name).toBe('name'); + expect(parsed.labels.dep_type).toBe('type'); + }); + + it('should accept empty metrics and labels (valid)', () => { + const result = validateMetricSchemaConfig({ metrics: {}, labels: {} }); + const parsed = JSON.parse(result); + expect(parsed.metrics).toEqual({}); + expect(parsed.labels).toEqual({}); + }); + + it('should reject invalid metric target field', () => { + expect(() => validateMetricSchemaConfig({ + metrics: { my_metric: 'invalid_target' }, + labels: {}, + })).toThrow(/invalid target "invalid_target"/); + }); + + it('should reject invalid label target field', () => { + expect(() => validateMetricSchemaConfig({ + metrics: {}, + labels: { my_label: 'invalid_target' }, + })).toThrow(/invalid target "invalid_target"/); + }); + + it('should reject duplicate target values in metrics', () => { + expect(() => validateMetricSchemaConfig({ + metrics: { metric_a: 'state', metric_b: 'state' }, + labels: {}, + })).toThrow(/duplicate target "state"/); + }); + + it('should reject duplicate target values in labels', () => { + expect(() => validateMetricSchemaConfig({ + metrics: {}, + labels: { label_a: 'name', label_b: 'name' }, + })).toThrow(/duplicate target "name"/); + }); + + it('should accept valid latency_unit', () => { + const result = validateMetricSchemaConfig({ metrics: {}, labels: {}, latency_unit: 's' }); + const parsed = JSON.parse(result); + expect(parsed.latency_unit).toBe('s'); + }); + + it('should reject invalid latency_unit', () => { + expect(() => validateMetricSchemaConfig({ + metrics: {}, + labels: {}, + latency_unit: 'hours', + })).toThrow(/latency_unit must be "ms" or "s"/); + }); + + it('should reject missing metrics key', () => { + expect(() => validateMetricSchemaConfig({ + labels: {}, + })).toThrow(/schema_config\.metrics is required/); + }); + + it('should reject missing labels key', () => { + expect(() => validateMetricSchemaConfig({ + metrics: {}, + })).toThrow(/schema_config\.labels is required/); + }); + + it('should accept JSON string input', () => { + const result = validateMetricSchemaConfig(JSON.stringify(validConfig)); + const parsed = JSON.parse(result); + expect(parsed.metrics.http_status).toBe('state'); + }); + + it('should reject non-object input', () => { + expect(() => validateMetricSchemaConfig(123)).toThrow(ValidationError); + expect(() => validateMetricSchemaConfig(true)).toThrow(ValidationError); + }); + + it('should reject array input', () => { + expect(() => validateMetricSchemaConfig([1, 2])).toThrow(ValidationError); + }); + + it('should reject invalid JSON string', () => { + expect(() => validateMetricSchemaConfig('not-json')).toThrow(ValidationError); + }); + + it('should reject non-string metric target values', () => { + expect(() => validateMetricSchemaConfig({ + metrics: { my_metric: 123 }, + labels: {}, + })).toThrow(/must be a string/); + }); + + it('should reject non-string label target values', () => { + expect(() => validateMetricSchemaConfig({ + metrics: {}, + labels: { my_label: 42 }, + })).toThrow(/must be a string/); + }); + + it('should omit latency_unit from output when not provided', () => { + const result = validateMetricSchemaConfig({ metrics: {}, labels: {} }); + const parsed = JSON.parse(result); + expect(parsed.latency_unit).toBeUndefined(); + }); + + it('should accept all valid metric targets', () => { + const config = { + metrics: { + m1: 'state', + m2: 'healthy', + m3: 'latency', + m4: 'code', + m5: 'skipped', + }, + labels: {}, + }; + const result = validateMetricSchemaConfig(config); + const parsed = JSON.parse(result); + expect(Object.values(parsed.metrics)).toEqual(['state', 'healthy', 'latency', 'code', 'skipped']); + }); + + it('should accept all valid label targets', () => { + const config = { + metrics: {}, + labels: { + l1: 'name', + l2: 'type', + l3: 'impact', + l4: 'description', + l5: 'errorMessage', + }, + }; + const result = validateMetricSchemaConfig(config); + const parsed = JSON.parse(result); + expect(Object.values(parsed.labels)).toEqual(['name', 'type', 'impact', 'description', 'errorMessage']); + }); + }); + + describe('format-aware validateServiceCreate with MetricSchemaConfig', () => { + const metricConfig = { + metrics: { http_status: 'state' }, + labels: { dep_name: 'name' }, + }; + + const schemaMappingConfig = { + root: 'data.checks', + fields: { name: 'n', healthy: 'h' }, + }; + + it('should accept MetricSchemaConfig with format=prometheus', () => { + const result = validateServiceCreate({ + name: 'Prom Service', + team_id: 'team-1', + health_endpoint: 'https://example.com/metrics', + health_endpoint_format: 'prometheus', + schema_config: metricConfig, + }); + expect(result.schema_config).toBeDefined(); + const parsed = JSON.parse(result.schema_config!); + expect(parsed.metrics.http_status).toBe('state'); + }); + + it('should reject SchemaMapping-shaped config with format=prometheus', () => { + expect(() => validateServiceCreate({ + name: 'Prom Service', + team_id: 'team-1', + health_endpoint: 'https://example.com/metrics', + health_endpoint_format: 'prometheus', + schema_config: schemaMappingConfig, + })).toThrow(ValidationError); + }); + + it('should set schema_config to null for format=default', () => { + const result = validateServiceCreate({ + name: 'Default Service', + team_id: 'team-1', + health_endpoint: 'https://example.com/health', + health_endpoint_format: 'default', + schema_config: { metrics: {}, labels: {} }, + }); + expect(result.schema_config).toBeNull(); + }); + + it('should set schema_config to null for format=default even without schema_config input', () => { + const result = validateServiceCreate({ + name: 'Default Service', + team_id: 'team-1', + health_endpoint: 'https://example.com/health', + health_endpoint_format: 'default', + }); + expect(result.schema_config).toBeNull(); + }); + + it('should use SchemaMapping validator when format=schema', () => { + const result = validateServiceCreate({ + name: 'Schema Service', + team_id: 'team-1', + health_endpoint: 'https://example.com/health', + health_endpoint_format: 'schema', + schema_config: schemaMappingConfig, + }); + expect(result.schema_config).toBeDefined(); + const parsed = JSON.parse(result.schema_config!); + expect(parsed.root).toBe('data.checks'); + }); + + it('should use SchemaMapping validator when format is undefined', () => { + const result = validateServiceCreate({ + name: 'No Format Service', + team_id: 'team-1', + health_endpoint: 'https://example.com/health', + schema_config: schemaMappingConfig, + }); + expect(result.schema_config).toBeDefined(); + const parsed = JSON.parse(result.schema_config!); + expect(parsed.root).toBe('data.checks'); + }); + }); + + describe('format-aware validateServiceUpdate with MetricSchemaConfig', () => { + const metricConfig = { + metrics: { otel_status: 'healthy' }, + labels: { service_name: 'name' }, + }; + + it('should accept MetricSchemaConfig with format=otlp', () => { + const result = validateServiceUpdate({ + health_endpoint_format: 'otlp', + schema_config: metricConfig, + }); + expect(result).not.toBeNull(); + expect(result!.schema_config).toBeDefined(); + const parsed = JSON.parse(result!.schema_config!); + expect(parsed.metrics.otel_status).toBe('healthy'); + }); + + it('should accept MetricSchemaConfig with format=prometheus in update', () => { + const result = validateServiceUpdate({ + health_endpoint_format: 'prometheus', + schema_config: metricConfig, + }); + expect(result).not.toBeNull(); + const parsed = JSON.parse(result!.schema_config!); + expect(parsed.labels.service_name).toBe('name'); + }); + + it('should auto-detect MetricSchemaConfig shape when format is not in payload', () => { + const result = validateServiceUpdate({ + schema_config: metricConfig, + }); + expect(result).not.toBeNull(); + const parsed = JSON.parse(result!.schema_config!); + expect(parsed.metrics.otel_status).toBe('healthy'); + }); + + it('should auto-detect SchemaMapping shape when format is not in payload', () => { + const result = validateServiceUpdate({ + schema_config: { + root: 'checks', + fields: { name: 'n', healthy: 'h' }, + }, + }); + expect(result).not.toBeNull(); + const parsed = JSON.parse(result!.schema_config!); + expect(parsed.root).toBe('checks'); + }); + + it('should set schema_config to null for format=default in update', () => { + const result = validateServiceUpdate({ + health_endpoint_format: 'default', + schema_config: { metrics: {}, labels: {} }, + }); + expect(result).not.toBeNull(); + expect(result!.schema_config).toBeNull(); + }); + }); +}); + describe('External Service Validation', () => { describe('validateExternalServiceCreate', () => { it('should validate valid input', () => { diff --git a/server/src/utils/validation.ts b/server/src/utils/validation.ts index 82a05f2..1113543 100644 --- a/server/src/utils/validation.ts +++ b/server/src/utils/validation.ts @@ -1,7 +1,9 @@ import { ValidationError } from './errors'; -import { AssociationType, DependencyType, TeamMemberRole, SchemaMapping, FieldMapping } from '../db/types'; +import { AssociationType, DependencyType, TeamMemberRole, SchemaMapping, FieldMapping, HealthEndpointFormat, MetricSchemaConfig, VALID_METRIC_TARGETS, VALID_LABEL_TARGETS } from '../db/types'; import { validateUrlHostname } from './ssrf'; +const VALID_HEALTH_ENDPOINT_FORMATS: HealthEndpointFormat[] = ['default', 'schema', 'prometheus', 'otlp']; + // ============================================================================ // URL Validation (moved from routes/services/validation.ts) // ============================================================================ @@ -121,6 +123,7 @@ export interface ValidatedServiceInput { metrics_endpoint: string | null; schema_config?: string | null; poll_interval_ms?: number; + health_endpoint_format?: HealthEndpointFormat; } export interface ValidatedServiceUpdateInput { @@ -131,6 +134,7 @@ export interface ValidatedServiceUpdateInput { schema_config?: string | null; poll_interval_ms?: number; is_active?: boolean; + health_endpoint_format?: HealthEndpointFormat; } /** @@ -148,12 +152,33 @@ export function validateServiceCreate(input: Record): Validated throw new ValidationError('team_id is required', 'team_id'); } - // Required: health_endpoint - if (!isString(input.health_endpoint) || !input.health_endpoint) { - throw new ValidationError('health_endpoint is required', 'health_endpoint'); + // Optional: health_endpoint_format (defaults to 'default') + let healthEndpointFormat: HealthEndpointFormat | undefined; + if (input.health_endpoint_format !== undefined) { + if (!isString(input.health_endpoint_format) || + !VALID_HEALTH_ENDPOINT_FORMATS.includes(input.health_endpoint_format as HealthEndpointFormat)) { + throw new ValidationError( + `health_endpoint_format must be one of: ${VALID_HEALTH_ENDPOINT_FORMATS.join(', ')}`, + 'health_endpoint_format' + ); + } + healthEndpointFormat = input.health_endpoint_format as HealthEndpointFormat; } - validateEndpointUrl(input.health_endpoint, 'health_endpoint'); + const isOtlp = healthEndpointFormat === 'otlp'; + + // health_endpoint: required for polled formats, empty for OTLP + if (isOtlp) { + // OTLP is push-only — health_endpoint not used + if (input.health_endpoint && isNonEmptyString(input.health_endpoint)) { + throw new ValidationError('health_endpoint must be empty for OTLP format (push-only)', 'health_endpoint'); + } + } else { + if (!isString(input.health_endpoint) || !input.health_endpoint) { + throw new ValidationError('health_endpoint is required', 'health_endpoint'); + } + validateEndpointUrl(input.health_endpoint, 'health_endpoint'); + } // Optional: metrics_endpoint let metricsEndpoint: string | null = null; @@ -167,9 +192,11 @@ export function validateServiceCreate(input: Record): Validated metricsEndpoint = input.metrics_endpoint || null; } - // Optional: poll_interval_ms + // Optional: poll_interval_ms (not applicable for OTLP) let pollIntervalMs: number | undefined; - if (input.poll_interval_ms !== undefined) { + if (isOtlp) { + pollIntervalMs = 0; + } else if (input.poll_interval_ms !== undefined) { if (!isNumber(input.poll_interval_ms) || !Number.isInteger(input.poll_interval_ms)) { throw new ValidationError('poll_interval_ms must be an integer', 'poll_interval_ms'); } @@ -182,23 +209,31 @@ export function validateServiceCreate(input: Record): Validated pollIntervalMs = input.poll_interval_ms; } - // Optional: schema_config + // Optional: schema_config (format-aware) let schemaConfig: string | null | undefined; if (input.schema_config !== undefined) { if (input.schema_config === null) { schemaConfig = null; + } else if (healthEndpointFormat === 'prometheus' || healthEndpointFormat === 'otlp') { + schemaConfig = validateMetricSchemaConfig(input.schema_config); + } else if (healthEndpointFormat === 'default') { + schemaConfig = null; } else { + // 'schema' or undefined — use original schema validator schemaConfig = validateSchemaConfig(input.schema_config); } + } else if (healthEndpointFormat === 'default') { + schemaConfig = null; } return { name: input.name.trim(), team_id: input.team_id, - health_endpoint: input.health_endpoint, + health_endpoint: isOtlp ? '' : (input.health_endpoint as string), metrics_endpoint: metricsEndpoint, schema_config: schemaConfig, poll_interval_ms: pollIntervalMs, + health_endpoint_format: healthEndpointFormat, }; } @@ -262,12 +297,42 @@ export function validateServiceUpdate( hasUpdates = true; } - // Optional: schema_config + // Optional: schema_config (format-aware) if (input.schema_config !== undefined) { if (input.schema_config === null) { result.schema_config = null; } else { - result.schema_config = validateSchemaConfig(input.schema_config); + const updateFormat = input.health_endpoint_format as HealthEndpointFormat | undefined; + if (updateFormat === 'prometheus' || updateFormat === 'otlp') { + result.schema_config = validateMetricSchemaConfig(input.schema_config); + } else if (updateFormat === 'default') { + result.schema_config = null; + } else if (updateFormat === 'schema' || updateFormat === undefined) { + // Format not in payload — detect config shape + if (updateFormat === undefined) { + let parsed: Record; + if (isString(input.schema_config)) { + try { + parsed = JSON.parse(input.schema_config) as Record; + } catch { + // Let the validators produce the proper error + parsed = {}; + } + } else if (typeof input.schema_config === 'object' && input.schema_config !== null) { + parsed = input.schema_config as Record; + } else { + parsed = {}; + } + if ('metrics' in parsed || 'labels' in parsed) { + result.schema_config = validateMetricSchemaConfig(input.schema_config); + } else { + result.schema_config = validateSchemaConfig(input.schema_config); + } + } else { + // updateFormat === 'schema' + result.schema_config = validateSchemaConfig(input.schema_config); + } + } } hasUpdates = true; } @@ -281,6 +346,19 @@ export function validateServiceUpdate( hasUpdates = true; } + // Optional: health_endpoint_format + if (input.health_endpoint_format !== undefined) { + if (!isString(input.health_endpoint_format) || + !VALID_HEALTH_ENDPOINT_FORMATS.includes(input.health_endpoint_format as HealthEndpointFormat)) { + throw new ValidationError( + `health_endpoint_format must be one of: ${VALID_HEALTH_ENDPOINT_FORMATS.join(', ')}`, + 'health_endpoint_format' + ); + } + result.health_endpoint_format = input.health_endpoint_format as HealthEndpointFormat; + hasUpdates = true; + } + return hasUpdates ? result : null; } @@ -595,6 +673,134 @@ export function validateDependencyType(type: unknown): DependencyType { return type as DependencyType; } +// ============================================================================ +// Metric Schema Config Validation (Prometheus / OTLP) +// ============================================================================ + +/** + * Validate and serialize a MetricSchemaConfig value for Prometheus/OTLP formats. + * Accepts either a JSON string or an object. Returns a validated JSON string. + * @throws ValidationError if the config is invalid + */ +export function validateMetricSchemaConfig(value: unknown): string { + let parsed: unknown; + + if (isString(value)) { + try { + parsed = JSON.parse(value); + } catch { + throw new ValidationError('schema_config must be valid JSON', 'schema_config'); + } + } else if (typeof value === 'object' && value !== null) { + parsed = value; + } else { + throw new ValidationError( + 'schema_config must be a JSON string or object', + 'schema_config' + ); + } + + if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) { + throw new ValidationError('schema_config must be a JSON object', 'schema_config'); + } + + const config = parsed as Record; + + // Validate metrics (required object) + if (!('metrics' in config)) { + throw new ValidationError('schema_config.metrics is required', 'schema_config'); + } + if (typeof config.metrics !== 'object' || config.metrics === null || Array.isArray(config.metrics)) { + throw new ValidationError('schema_config.metrics must be an object', 'schema_config'); + } + + const metrics = config.metrics as Record; + const seenMetricTargets = new Set(); + for (const [key, target] of Object.entries(metrics)) { + if (!isString(target)) { + throw new ValidationError( + `schema_config.metrics.${key} must be a string`, + 'schema_config' + ); + } + if (!VALID_METRIC_TARGETS.includes(target as typeof VALID_METRIC_TARGETS[number])) { + throw new ValidationError( + `schema_config.metrics.${key} has invalid target "${target}". Valid targets: ${VALID_METRIC_TARGETS.join(', ')}`, + 'schema_config' + ); + } + if (seenMetricTargets.has(target)) { + throw new ValidationError( + `schema_config.metrics has duplicate target "${target}"`, + 'schema_config' + ); + } + seenMetricTargets.add(target); + } + + // Validate labels (required object) + if (!('labels' in config)) { + throw new ValidationError('schema_config.labels is required', 'schema_config'); + } + if (typeof config.labels !== 'object' || config.labels === null || Array.isArray(config.labels)) { + throw new ValidationError('schema_config.labels must be an object', 'schema_config'); + } + + const labels = config.labels as Record; + const seenLabelTargets = new Set(); + for (const [key, target] of Object.entries(labels)) { + if (!isString(target)) { + throw new ValidationError( + `schema_config.labels.${key} must be a string`, + 'schema_config' + ); + } + if (!VALID_LABEL_TARGETS.includes(target as typeof VALID_LABEL_TARGETS[number])) { + throw new ValidationError( + `schema_config.labels.${key} has invalid target "${target}". Valid targets: ${VALID_LABEL_TARGETS.join(', ')}`, + 'schema_config' + ); + } + if (seenLabelTargets.has(target)) { + throw new ValidationError( + `schema_config.labels has duplicate target "${target}"`, + 'schema_config' + ); + } + seenLabelTargets.add(target); + } + + // Validate latency_unit (optional) + if (config.latency_unit !== undefined) { + if (config.latency_unit !== 'ms' && config.latency_unit !== 's') { + throw new ValidationError( + 'schema_config.latency_unit must be "ms" or "s"', + 'schema_config' + ); + } + } + + // Validate healthy_value (optional) + if (config.healthy_value !== undefined) { + if (!isNumber(config.healthy_value)) { + throw new ValidationError( + 'schema_config.healthy_value must be a number', + 'schema_config' + ); + } + } + + // Build validated object + const validated: MetricSchemaConfig = { + metrics: metrics as Record, + labels: labels as Record, + ...(config.latency_unit !== undefined && { latency_unit: config.latency_unit as 'ms' | 's' }), + ...(config.healthy_value !== undefined && { healthy_value: config.healthy_value as number }), + }; + + return JSON.stringify(validated); +} + // ============================================================================ // Schema Config Validation // ============================================================================