Build products that see, hear, and understand — with zero cloud lock-in
Quick Start · Demo · Architecture · SDKs · Examples · Docs
Sense Platform is a complete, self-hosted alternative to GetStream — rebuilt from scratch for AI-first products. Every component runs on your own servers with a single docker compose up.
|
What you replace
|
What you get instead
|
Run the interactive demo against your local stack — or in simulated mode without any services running:
# With the platform running
docker compose up -d
python demo.py
# Without the platform (simulated output)
python demo.pyWhat the demo walks through:
Step 1 Health-check all 5 services
Step 2 Authenticate with Sense Gate (API key → JWT)
Step 3 Create a demo room
Step 4 Start an AI agent with MoodLens + FaceLens
Step 5 Send real-time messages via Sense Wire
Step 6 Stream live vision lens events
Step 7 Reconfigure MoodLens throttle at runtime
Step 8 Stop the agent and clean up
Sample output:
███████╗███████╗███╗ ██╗███████╗███████╗
██╔════╝██╔════╝████╗ ██║██╔════╝██╔════╝
███████╗█████╗ ██╔██╗ ██║███████╗█████╗
╚════██║██╔══╝ ██║╚██╗██║╚════██║██╔══╝
███████║███████╗██║ ╚████║███████║███████╗
╚══════╝╚══════╝╚═╝ ╚═══╝╚══════╝╚══════╝
AI-powered video · voice · vision — self-hosted
Step 1 Health-checking all services
Sense Gate :3000 [LIVE] status=ok
Sense Wire :3001 [LIVE] status=ok
Sense Mind :8080 [LIVE] status=ok
Sense Relay :7880 [LIVE] status=ok
Step 2 Authenticating with Sense Gate
✓ Authenticated [real JWT]
Step 4 Starting AI agent (MoodLens + FaceLens)
✓ Agent started [live]
room = acme__demo-room
llm = claude-sonnet-4-6
lenses = ['MoodLens', 'FaceLens']
Step 6 Streaming vision lens events
[14:03:01] MOOD: NEUTRAL conf=91%
context → Participant appears calm and attentive.
[14:03:01] FACE: 1 detected conf=98%
context → 1 face detected in frame.
[14:03:02] MOOD: CURIOUS conf=83%
context → Participant is engaged and leaning forward.
[14:03:03] MOOD: SATISFIED conf=87%
context → Participant appears satisfied with the response.
┌──────────────────────────────────────┐
Browser / SDK ────►│ Sense Gate :3000 │◄──── Sense Console :4000
Client ◄────│ REST · Auth · Multi-tenancy │ Developer Dashboard
└─────────────┬──────────────┬──────────┘
│ │
┌────────────────────┘ └────────────────────┐
│ │
┌──────────▼──────────┐ ┌──────────────▼──────────┐
│ Sense Mind :8080 │ │ Sense Wire :3001 │
│ ┌────────────────┐ │ ── LensEventBridge ────────►│ WebSocket Messaging │
│ │ AI Agents │ │ │ Redis pub/sub fan-out │
│ │ + AgentPool │ │ └─────────────────────────┘
│ └────────────────┘ │
│ ┌────────────────┐ │
│ │ MoodLens │ │
│ │ PoseLens │ │
│ │ GuardLens │ │
│ │ FaceLens │ │
│ └────────────────┘ │
└──────────┬───────────┘
│
┌──────────▼──────────┐ ┌──────────────────────────┐
│ Sense Relay :7880 │ │ Postgres + Redis │
│ WebRTC SFU │ │ data · state · pub/sub │
│ (LiveKit engine) │ └──────────────────────────┘
└─────────────────────┘
Services at a glance
| Service | Role | Port |
|---|---|---|
| Sense Relay | Self-hosted WebRTC SFU — audio/video routing, no cloud | 7880 |
| Sense Mind | AI agent engine — LLM · STT · TTS · Vision Lenses · AgentPool | 8080 |
| Sense Gate | REST API gateway — JWT auth, multi-tenancy, webhooks, usage metering | 3000 |
| Sense Wire | Real-time WebSocket messaging — Redis pub/sub cross-instance fan-out | 3001 |
| Sense Console | Next.js developer dashboard — rooms, agents, keys, webhooks, usage | 4000 |
Vision Lenses
Lenses are throttled vision processors that inject context directly into the agent's LLM prompt mid-conversation — no prompt engineering required.
| Lens | What it detects | Throttle |
|---|---|---|
| MoodLens | Emotion — frustrated · confused · satisfied · happy | 3 s |
| PoseLens | Body keypoints — posture, form faults, gestures | 2 s |
| GuardLens | Safety — weapons, explicit content, fire/smoke, tailgating | 2 s |
| FaceLens | Presence — face count, occupancy, identity continuity | 5 s |
Prerequisites: Docker Desktop · one LLM API key (Anthropic recommended)
git clone https://github.com/VenkataAnilKumar/SensePlatform
cd SensePlatform
cp .env.example .envAdd your key to .env:
ANTHROPIC_API_KEY=sk-ant-...docker compose upAll five services start automatically. First run pulls images (~2 min). After ~30 s:
| URL | Service |
|---|---|
| http://localhost:4000 | Sense Console — developer dashboard |
| http://localhost:3000 | Sense Gate — REST API |
| ws://localhost:3001 | Sense Wire — WebSocket |
| http://localhost:8080 | Sense Mind — agent control |
| ws://localhost:7880 | Sense Relay — WebRTC |
python demo.pycurl -s -X POST http://localhost:3000/agents/start \
-H "X-API-Key: sk_live_your_key" \
-H "Content-Type: application/json" \
-d '{
"room_id": "demo",
"lenses": ["MoodLens"],
"llm": "claude-sonnet-4-6",
"instructions": "You are a helpful assistant. Adapt to the user emotional state."
}' | jq .pip install sense-platformimport asyncio
from sense import SenseClient, SenseWire, LensStream
async def main():
async with SenseClient(api_key="sk_live_...") as client:
await client.connect()
# Launch AI agent with emotion detection
await client.agents.start("demo", lenses=["MoodLens"])
# Subscribe to messages + live lens events
async with SenseWire(client, "messaging", "demo") as wire:
stream = LensStream(wire)
@stream.on_mood
def on_mood(event):
print(f"[{event.mood}] {event.context_text}")
wire.on_message(lambda msg: print(f"{msg.user_id}: {msg.text}"))
await wire.send("Hello from Python!")
await wire.wait()
asyncio.run(main())npm install @sense/core @sense/vision @sense/chat @sense/lensimport { SenseClient } from "@sense/core";
import { SenseRoom } from "@sense/vision";
import { Channel } from "@sense/chat";
import { LensStream } from "@sense/lens";
const client = new SenseClient({ apiKey: "sk_live_..." });
await client.connect();
// Join WebRTC room (publishes webcam + mic)
const room = new SenseRoom(client);
await room.join({ roomId: "demo", autoPublish: true });
// Chat
const channel = new Channel(client.wire, "messaging", "demo");
await channel.sendMessage("Hello from the browser!");
// Live vision events
const lenses = new LensStream(client.wire, "demo");
lenses.onMood((e) => console.log(`${e.mood} — ${e.contextText}`));
lenses.onGuard((e) => { if (e.violation) alert(`⚠ ${e.violationType}`); });Sense Mind Agent SDK — build custom agents
from sense_mind import SenseMind, SenseRunner
from sense_mind.plugins import anthropic, deepgram, elevenlabs
from sense_mind.lenses import MoodLens, PoseLens
agent = SenseMind(
instructions="""
You are a supportive contact center agent.
When the customer is frustrated, acknowledge it first.
When confused, slow down and use plain language.
""",
llm=anthropic.LLM("claude-sonnet-4-6"),
stt=deepgram.STT(),
tts=elevenlabs.TTS(),
lenses=[MoodLens(throttle_seconds=3), PoseLens()],
)
SenseRunner(agent, room="support-room-1").serve()Supported plugins
| Category | Options |
|---|---|
| LLM | anthropic · openai · gemini · mistral · openrouter · xai |
| STT | deepgram · assemblyai · fast_whisper |
| TTS | elevenlabs · cartesia · kokoro · fish · pocket |
| Vision | ultralytics (YOLO) · moondream · roboflow · nvidia |
Sense Mind manages multiple agents concurrently — one per room, controlled via REST:
# Launch an agent
curl -X POST http://localhost:8080/agents/start \
-H "Content-Type: application/json" \
-d '{"room": "acme__room-1", "lenses": ["MoodLens"], "llm": "claude-sonnet-4-6"}'
# Check all running agents
curl http://localhost:8080/agents/status
# Tune lens throttle at runtime — zero downtime
curl -X POST http://localhost:8080/agents/acme__room-1/lenses/mood_lens/configure \
-d '{"throttle_seconds": 5.0, "enabled": true}'
# Stop an agent
curl -X POST http://localhost:8080/agents/stop \
-d '{"room": "acme__room-1"}'Five production-grade examples in examples/:
| Example | Lenses | What it does |
|---|---|---|
contact-center/ |
MoodLens · FaceLens | Adapts tone to frustration; escalates to human on trigger |
sales-coach/ |
MoodLens · PoseLens | Silent whisper coach — spots buying signals, flags objections |
fitness-coach/ |
PoseLens · FaceLens | Real-time form cues; rep counting; rest detection |
telehealth/ |
FaceLens · PoseLens · MoodLens | Clinical assistant — logs observations, flags distress |
security-soc/ |
FaceLens · GuardLens | Zone monitoring; weapon/threat detection; auto-incident |
Run any example:
# Contact center
ANTHROPIC_API_KEY=sk-ant-... SENSE_ROOM=support python examples/contact-center/agent.py
# Fitness coach (squats)
EXERCISE=squat SENSE_ROOM=gym-alice python examples/fitness-coach/agent.py
# Security SOC — restricted zone
ZONE_TYPE=restricted ZONE_ID=server-room-a python examples/security-soc/agent.py| Quick Start | Up and running in 5 minutes |
| Architecture | How the five services connect, data flow, multi-tenancy |
| Self-Hosting Guide | TLS, backups, horizontal scaling, production checklist |
| Python SDK Reference | Full API reference for every class and method |
| Contact Center Tutorial | End-to-end product guide with React frontend + Python supervisor |
All eight phases shipped.
| Phase | What shipped | |
|---|---|---|
| ✅ | 1 — Core | Sense Relay · Sense Mind · Vision Lenses · Docker Compose |
| ✅ | 2 — Gate | FastAPI gateway · JWT auth · multi-tenancy · webhooks · usage metering |
| ✅ | 3 — Wire | WebSocket messaging · Postgres persistence · Redis pub/sub fan-out |
| ✅ | 4 — JS SDK | @sense/core · @sense/vision · @sense/chat · @sense/lens |
| ✅ | 5 — Console | Next.js developer dashboard — rooms, agents, keys, webhooks, usage |
| ✅ | 6 — Agent Pool | Multi-agent pool · LensEventBridge · Vision Event Stream · /agents API |
| ✅ | 7 — Py SDK + Docs | sense-platform Python SDK · quickstart · architecture · self-hosting guide |
| ✅ | 8 — Examples | Contact center · sales coach · fitness coach · telehealth · security SOC |
Apache 2.0 · Built with LiveKit, FastAPI, Next.js, and Claude