diff --git a/AGENTS.md b/AGENTS.md index 17984d7..cddf783 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,12 +34,15 @@ Backend config is loaded from environment variables (see `backend/.env.example`) - `CLERK_WEBHOOK_SECRET` — Svix signature for Clerk subscription webhooks - `DATABASE_URL` — defaults to `sqlite:///./opensentry.db` - `FRONTEND_URL` — extra CORS origin (must have scheme, no trailing slash) -- `SEGMENT_CACHE_MAX_PER_CAMERA` — segments cached in memory per camera (default 15, ~30s) +- `SEGMENT_CACHE_MAX_PER_CAMERA` — segments cached in memory per camera (default 60, ~60s) - `SEGMENT_PUSH_MAX_BYTES` — max bytes per pushed segment (default 2 MB) - `CLEANUP_INTERVAL` — run cache eviction every N playlist updates (default 20) - `INACTIVE_CAMERA_CLEANUP_HOURS` — free caches for cameras offline this long (default 24) - `LOG_RETENTION_DAYS` — stream + MCP + audit + motion + notification log retention (default 90) - `OFFLINE_SWEEP_INTERVAL_SECONDS` — how often to mark stale rows offline (default 30) +- `REDIS_URL` — Redis connection for rate limiter storage; in-memory fallback when unset (fine for single-instance, but limits don't hold across VMs) +- `SENTRY_DSN` — Sentry error tracking DSN; leave blank to disable (no-ops gracefully) +- `SENTRY_TRACES_SAMPLE_RATE` — Sentry performance trace sample rate (default 0.1) Frontend config: `VITE_CLERK_PUBLISHABLE_KEY`, `VITE_API_URL`, `VITE_LOCAL_HLS`. @@ -64,13 +67,20 @@ backend/ │ │ ├── ws.py # CloudNode WebSocket channel │ │ └── webhooks.py # Clerk subscription webhook handler │ ├── mcp/ -│ │ └── server.py # FastMCP server + 22 tools + ScopeMiddleware +│ │ ├── server.py # FastMCP server + 22 tools + ScopeMiddleware +│ │ └── activity.py # MCP activity tracking + per-org broadcasting │ ├── core/ │ │ ├── auth.py # Clerk JWT validation (V1 + V2 permissions), dependencies +│ │ ├── audit.py # Audit log helper (records admin actions) +│ │ ├── codec.py # Video codec string sanitization │ │ ├── config.py # Environment loading (Config class) │ │ ├── clerk.py # Clerk SDK init │ │ ├── database.py # SQLAlchemy engine + session factory + Base -│ │ └── limiter.py # slowapi Limiter instance (tenant-aware key) +│ │ ├── limiter.py # slowapi Limiter instance (tenant-aware key) +│ │ ├── migrations.py # Manual schema migrations (stand-in for Alembic) +│ │ ├── plans.py # Plan limit logic (node quotas per tier) +│ │ ├── sentry.py # Sentry error tracking initialization +│ │ └── versions.py # CloudNode version compatibility checking │ ├── models/models.py # 13 ORM models (see Data Models below) │ └── schemas/schemas.py # Pydantic request/response schemas incl. McpKeyCreate ├── scripts/ @@ -99,6 +109,7 @@ frontend/ │ ├── HlsPlayer.jsx # HLS.js player with Clerk JWT xhrSetup │ ├── CameraCard.jsx # Live thumbnail + status + actions │ ├── CameraGridPreview.jsx # Static preview for the landing page + │ ├── DocsDiagrams.jsx # SVG architecture diagrams for the docs page │ ├── IncidentReportModal.jsx # Markdown + evidence viewer │ ├── NotificationBell.jsx # Unread badge + inbox popover (SSE-fed) │ ├── AddNodeModal.jsx # Node creation flow (shows one-time API key) @@ -139,7 +150,7 @@ MCP Client ──Bearer osc_…──→ FastMCP → ScopeMiddleware → tools ### Video streaming pipeline -1. CloudNode generates HLS segments via FFmpeg (2-second `.ts` files) +1. CloudNode generates HLS segments via FFmpeg (1-second `.ts` files) 2. CloudNode calls `POST /api/cameras/{id}/push-segment?filename=segment_NNNNN.ts` with the raw `.ts` body and `X-Node-API-Key` header 3. Backend stores the bytes in `_segment_cache[camera_id][filename]`, evicting the oldest once `SEGMENT_CACHE_MAX_PER_CAMERA` is exceeded 4. CloudNode calls `POST /api/cameras/{id}/playlist` with the rolling `stream.m3u8` text @@ -151,7 +162,7 @@ MCP Client ──Bearer osc_…──→ FastMCP → ScopeMiddleware → tools ### SPA serving `main.py` SPA middleware: -- `/api/*`, `/ws/*`, `/install.*`, `/mcp-setup.*` → FastAPI handlers +- `/api/*`, `/ws/*`, `/install.*`, `/mcp-setup.*`, `/downloads/*` → FastAPI handlers - `POST /mcp` → FastMCP ASGI app (streamable HTTP) - `GET /mcp` → React `McpPage` (dashboard route) - `/assets/*` → static files from React build @@ -314,6 +325,7 @@ Validation constants (also in `models.py`): **install.py** (no prefix, no auth): - `GET /install.sh` / `GET /install.ps1` — CloudNode installer scripts - `GET /mcp-setup.sh` / `GET /mcp-setup.ps1` — MCP client config helpers +- `GET /downloads/{os_name}/{arch}` — 302 redirect to the latest CloudNode binary on GitHub Releases (os: `linux`/`macos`/`windows`, arch: `x86_64`/`aarch64`/`armv7`) **ws.py** (no prefix): - `WS /ws/node` — WebSocket channel for CloudNode realtime (API key in query) @@ -390,7 +402,7 @@ cors_origins = [ "https://opensentry-command.fly.dev", ] ``` -Plus `FRONTEND_URL` if set (validated: must have scheme, no trailing slash, no embedded whitespace). All methods and headers allowed; credentials allowed. +Plus `FRONTEND_URL` if set (validated: must have scheme, no trailing slash, no embedded whitespace). Allowed methods: `GET`, `POST`, `PUT`, `DELETE`, `OPTIONS`. Allowed headers: `Authorization`, `Content-Type`, `X-Node-API-Key`. Credentials allowed. ## Rate Limiting @@ -468,8 +480,11 @@ HLS `GET` paths (`stream.m3u8`, `segment/{file}`) are intentionally unlimited - `clerk-backend-api` — Clerk authentication - `pyjwt` — JWT token handling (for V2 permission decoding) - `slowapi` — Rate limiting +- `redis` — Rate limiter backing store (production; in-memory fallback for dev) - `httpx` — HTTP client +- `websockets` — WebSocket protocol support (CloudNode channel) - `svix` — Webhook signature verification +- `sentry-sdk[fastapi]` — Error tracking and performance monitoring - `python-dotenv` — Environment variable loading ## Development Notes diff --git a/README.md b/README.md index a31077a..ddcb5b0 100644 --- a/README.md +++ b/README.md @@ -92,18 +92,18 @@ The scripts detect which clients you already have and merge an `opensentry` entr │ USB Camera │ │ FastAPI Backend │ │ React 19 │ │ ↓ │ │ │ │ │ │ FFmpeg (HLS) │──push─────→│ In-memory segment │←─GET───→│ HLS.js │ - │ │ segments │ cache (~15 segs/cam) │ proxy │ (video) │ + │ │ segments │ cache (~60 segs/cam) │ proxy │ (video) │ │ │──register─→│ SQLite / PostgreSQL │ URLs │ │ │ │──heartbeat→│ Clerk Auth │←─JWT───→│ Clerk Auth │ │ │──WS events │ FastMCP (/mcp) │←──SSE───│ Motion feed │ └──────────────┘ └───────────────────────┘ └──────────────┘ ``` -**Video pipeline:** CloudNode transcodes USB camera video into HLS segments and pushes each `.ts` file directly to the backend via `POST /api/cameras/{id}/push-segment`. The backend caches segments in memory (15 per camera by default, ~30s buffer) and serves them through the same-origin proxy at `GET /api/cameras/{id}/segment/{file}`. The rewritten playlist contains relative segment URLs, so the browser's Clerk JWT auth header is automatically attached. No S3, no presigned URLs, no third-party storage in the live path. +**Video pipeline:** CloudNode transcodes USB camera video into HLS segments and pushes each `.ts` file directly to the backend via `POST /api/cameras/{id}/push-segment`. The backend caches segments in memory (60 per camera by default, ~60s buffer) and serves them through the same-origin proxy at `GET /api/cameras/{id}/segment/{file}`. The rewritten playlist contains relative segment URLs, so the browser's Clerk JWT auth header is automatically attached. No S3, no presigned URLs, no third-party storage in the live path. **Authentication:** Clerk handles user sign-up, login, and organization management. The backend validates JWT tokens (V1 and V2 permission formats) and extracts organization-scoped permissions. CloudNodes authenticate with API keys (SHA-256 hashed in the database) passed via `X-Node-API-Key`. MCP clients authenticate with `Authorization: Bearer osc_...` keys (also hashed). -**Storage:** Live segments live in the backend's in-memory cache; they expire automatically once `SEGMENT_CACHE_MAX_PER_CAMERA` is exceeded. Recordings and snapshots live on the CloudNode itself. SQLite is used for development (`opensentry.db`); PostgreSQL for production. Incident snapshots and clips are stored inline on `IncidentEvidence.data` (LargeBinary) — evidence travels with the incident. +**Storage:** Live segments live in the backend's in-memory cache; they expire automatically once `SEGMENT_CACHE_MAX_PER_CAMERA` is exceeded. Recordings and snapshots live on the CloudNode itself in its encrypted SQLite database. SQLite is used for development on the Command Center (`opensentry.db`); PostgreSQL for production. Incident snapshots and clips are stored inline on `IncidentEvidence.data` (LargeBinary) — evidence travels with the incident. **Real-time:** CloudNodes maintain a WebSocket channel (`/ws/node`) used for commands, status, and motion events. The dashboard subscribes to SSE feeds for motion events (`/api/motion/events/stream`), notifications (`/api/notifications/stream`), and MCP activity (`/api/mcp/activity/stream`). @@ -120,12 +120,15 @@ The scripts detect which clients you already have and merge an `opensentry` entr | `CLERK_WEBHOOK_SECRET` | No | | Svix signature secret for Clerk webhooks | | `DATABASE_URL` | No | `sqlite:///./opensentry.db` | SQLAlchemy connection string | | `FRONTEND_URL` | No | `http://localhost:5173` | Extra CORS origin (must include scheme, no trailing slash) | -| `SEGMENT_CACHE_MAX_PER_CAMERA` | No | `15` | Segments cached in memory per camera (~2s each) | +| `SEGMENT_CACHE_MAX_PER_CAMERA` | No | `60` | Segments cached in memory per camera (~1s each) | | `SEGMENT_PUSH_MAX_BYTES` | No | `2097152` | Max bytes per pushed segment (2 MB) | | `CLEANUP_INTERVAL` | No | `20` | Run cache eviction every N playlist updates | | `INACTIVE_CAMERA_CLEANUP_HOURS` | No | `24` | Free caches for cameras offline this long | | `LOG_RETENTION_DAYS` | No | `90` | Stream, MCP, audit, motion, and notification log retention | | `OFFLINE_SWEEP_INTERVAL_SECONDS` | No | `30` | How often to flip stale `online` rows to `offline` | +| `REDIS_URL` | No | (empty) | Redis connection for rate limiter storage; in-memory fallback when unset (fine for single-instance, but limits don't hold across VMs) | +| `SENTRY_DSN` | No | (empty) | Sentry error tracking DSN; leave blank to disable (no-ops gracefully) | +| `SENTRY_TRACES_SAMPLE_RATE` | No | `0.1` | Sentry performance trace sample rate (0.0–1.0) | ### Frontend environment variables @@ -258,6 +261,7 @@ See [AGENTS.md](AGENTS.md) for the full per-tool list. |--------|----------|-------------| | GET | `/install.sh` / `/install.ps1` | CloudNode installer scripts | | GET | `/mcp-setup.sh` / `/mcp-setup.ps1` | MCP client setup helpers | +| GET | `/downloads/{os}/{arch}` | 302 redirect to the latest CloudNode binary on GitHub Releases (os: `linux`/`macos`/`windows`, arch: `x86_64`/`aarch64`/`armv7`) | ### System @@ -327,13 +331,20 @@ backend/ │ │ ├── ws.py # WebSocket channel (heartbeat, commands, motion events) │ │ └── webhooks.py # Clerk subscription webhooks │ ├── mcp/ -│ │ └── server.py # FastMCP server + 22 tools + ScopeMiddleware +│ │ ├── server.py # FastMCP server + 22 tools + ScopeMiddleware +│ │ └── activity.py # MCP activity tracking + per-org broadcasting │ ├── core/ │ │ ├── auth.py # Clerk JWT validation (V1 + V2), dependencies +│ │ ├── audit.py # Audit log helper (records admin actions) +│ │ ├── codec.py # Video codec string sanitization │ │ ├── config.py # Environment loading (Config class) │ │ ├── clerk.py # Clerk SDK initialization │ │ ├── database.py # SQLAlchemy engine + session factory -│ │ └── limiter.py # slowapi Limiter instance +│ │ ├── limiter.py # slowapi Limiter instance (tenant-aware key) +│ │ ├── migrations.py # Manual schema migrations (stand-in for Alembic) +│ │ ├── plans.py # Plan limit logic (node quotas per tier) +│ │ ├── sentry.py # Sentry error tracking initialization +│ │ └── versions.py # CloudNode version compatibility checking │ ├── models/models.py # 13 ORM models (see table above) │ └── schemas/schemas.py # Pydantic request/response schemas ├── scripts/ @@ -364,8 +375,9 @@ frontend/ │ # HeartbeatBanner (first-heartbeat polling after │ # node creation, localStorage-backed), │ # WelcomeHero (Admin / Member empty-state heroes), - │ # CameraGridPreview, EmptyState, PublicLayout, - │ # LandingNav, LandingFooter, LoadingSpinner + │ # CameraGridPreview, DocsDiagrams, EmptyState, + │ # PublicLayout, LandingNav, LandingFooter, + │ # LoadingSpinner ├── hooks/ # useNotifications, useMotionAlerts, usePlanInfo, │ # useSharedToken, useToasts └── services/api.js # Typed client for every backend endpoint @@ -408,7 +420,7 @@ Deployed on [Fly.io](https://fly.io) via GitHub Actions: 3. Live video segments are cached in the backend's process memory — no external storage 4. Clerk handles authentication (no user database needed) -Memory sizing: each camera uses ~3.75 MB of cache (`SEGMENT_CACHE_MAX_PER_CAMERA × ~250 KB per segment`). The default 1 GB Fly instance comfortably handles ~150 cameras with headroom. Bump `[[vm]] memory_mb` if you need more. +Memory sizing: each camera uses ~7.5 MB of cache (`SEGMENT_CACHE_MAX_PER_CAMERA × ~125 KB per 1-second segment`). The default 1 GB Fly instance comfortably handles ~130 cameras with headroom. Bump `[[vm]] memory_mb` if you need more. Production URL: [opensentry-command.fly.dev](https://opensentry-command.fly.dev) diff --git a/SECURITY.md b/SECURITY.md index ba35986..eb83383 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,6 +24,7 @@ Always run the latest version. The current release is tracked in `backend/pyproj | **Audit Logging** | Stream access tracked with user ID, IP, and user agent | | **Encrypted Storage** | CloudNode encrypts API key at rest with AES-256-GCM | | **Webhook Verification** | Clerk webhooks verified via Svix signature | +| **Security Headers** | All responses include `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin`, `Permissions-Policy: camera=(), microphone=(), geolocation=()`, and `Strict-Transport-Security` (HTTPS) | ## Reporting a Vulnerability diff --git a/backend/.env.example b/backend/.env.example index 3f1dcac..87108bc 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -25,3 +25,13 @@ INACTIVE_CAMERA_CLEANUP_HOURS=24 # Log retention (stream, MCP, and audit logs — days) LOG_RETENTION_DAYS=90 + +# Redis for rate limiter storage (production). +# In dev the empty default is fine — slowapi falls back to in-memory storage. +# In production set this to a managed instance (Upstash on Fly, etc.) so +# limits hold across VMs. +# REDIS_URL=redis://localhost:6379 + +# Sentry error tracking (leave blank for local dev — no-ops gracefully) +# SENTRY_DSN=https://...@sentry.io/... +# SENTRY_TRACES_SAMPLE_RATE=0.1