Offline-first, RAG-enforced, $ecure Local AI "Second Brain" with Dropbox Sync for data/corpus/ + secure, agentic functionality
CyClaw is a personal RAG (Retrieval-Augmented Generation) backend that:
- Answers questions exclusively from your local Markdown corpus — no internet by default
- Enforces every safety invariant via LangGraph topology — not prompts, not config flags, not discipline
- Maintains a persistent soul/personality layer (
soul.md) with SHA-256 drift detection, atomic evolution writes, and user-gated modification - Falls back to an external LLM only with explicit user confirmation in hybrid mode — Grok (xAI) or Claude (Anthropic), selected per-query, each independently triple-gated at config, env, and per-query level
- Exposes both a FastAPI HTTP gateway and an MCP server for Claude Desktop / Copilot Studio integration
- Ships optional, out-of-band operator layers for Dropbox corpus sync (
sync/) and agentic GitHub context / governed local workflows (agentic/,.claude/) — never imported into the request path, now also drivable from the browser terminal via governed Sync and Agentic consoles - Extends the agentic layer to local data (v1.8) with an opt-in filesystem connector (
agentic/fsconnect/— scoped reads + gated writes over local/SMB shares, TOCTOU-safe) and a read-only SQL connector (agentic/sqlconnect/— SELECT-only Postgres/MSSQL scaffold) — both disabled by default and out-of-band - Adds an optional NeMo Guardrails content-safety layer (v1.8,
guardrails/) that soft-importsnemoguardrailsand degrades to offline heuristic rails — defense-in-depth only, never a routing authority (graph topology stays the sole policy)
User Query (HTTP POST /query or MCMC tool call)
│
▼
┌─────────────────────────────────────────────────────┐
│ gate.py (FastAPI, 127.0.0.1:8787) │
│ • Rate limit (60 req/min per IP — RUNS FIRST) │
│ • Injection filter (sanitizer.py, config-driven) │
│ • Soul init (PersonalityManager closure) │
│ • Telemetry kill block (before any SDK import) │
└──────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ graph.py (LangGraph 9-node State Machine) │
│ │
│ [ENTRY] │
│ ↓ │
│ 1. retrieve (Chroma + BM25 + RRF fusion) │
│ ↓ │
│ 2. route_score (top_score >= 0.028 RRF?) │
│ ├─ YES ──→ 3. guardrail_input (offline rail; │
│ │ opt-in, pass-through when disabled) │
│ │ blocked ──→ 8. audit_logger │
│ │ passed ──→ 4. local_llm │
│ │ (LM Studio :1234) │
│ └─ NO ──→ 5. user_gate (needs_confirm=true) │
│ ├─ confirmed + hybrid ──→ │
│ │ 6. grok_fallback OR │
│ │ claude_fallback │
│ │ (selected per-query) │
│ └─ declined / offline ──→ │
│ 7. offline_best_effort │
│ ↓ (all paths converge) │
│ 8. audit_logger (SHA-256 + PII redact → jsonl) │
│ ↓ │
│ [END] │
└─────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ HybridRetriever (retrieval/hybrid_search.py) │
│ • ChromaDB (semantic, all-MiniLM-L6-v2, 384d) │
│ • BM25Okapi (keyword, Porter stemming) │
│ • RRF fusion (k=60, equal 1.0/1.0 weighting) │
│ • Per-chunk provenance metadata in every result │
└─────────────────────────────────────────────────────┘
flowchart TD
A(["🌐 Client\nHTTP POST /query\nor MCP tool call"])
A --> B
subgraph GATEWAY ["gate.py — FastAPI 127.0.0.1:8787"]
B["TrustedHostMiddleware\nHost header allowlist"]
B --> C["Rate Limiter\n60 req/min per IP"]
C --> D["Prompt Injection Filter\n33 patterns · config-driven · lru_cache"]
D --> E["Build GraphState\nquery + user_confirmed_online"]
end
E --> F
subgraph GRAPH ["graph.py — LangGraph 9-node State Machine"]
F(["① retrieve\nChroma + BM25 + RRF"])
F --> G["② route_by_score\ntop_score ≥ 0.028?"]
G -->|"YES — local context"| X["③ guardrail_input\noffline rail · opt-in\npass-through when disabled"]
X -->|"blocked"| L
X -->|"passed"| H["④ local_llm\nLM Studio :1234\nQwen2.5-7b"]
G -->|"NO — vault miss"| I["⑤ user_gate\nneeds_confirm = true"]
I -->|"confirmed=true + hybrid\n+ grok.enabled + provider=grok"| J["⑥ grok_fallback\nxAI grok-4.3\ntriple-gated"]
I -->|"confirmed=true + hybrid\n+ claude.enabled + provider=claude"| W["⑦ claude_fallback\nAnthropic claude-sonnet-5\ntriple-gated"]
I -->|"confirmed=false\nor offline mode"| K["⑧ offline_best_effort\nlocal LLM · no RAG gate"]
H --> L
J --> L
W --> L
K --> L
L(["⑨ audit_logger\nSHA-256 hash · PII redact\n→ logs/audit.jsonl"])
end
L --> M(["📤 QueryResponse\nanswer · sources · model_used\nretrieval_mode · needs_confirm"])
subgraph RETRIEVAL ["retrieval/hybrid_search.py"]
N["ChromaDB\nsemantic · 384-dim cosine"]
O["BM25Okapi\nkeyword · Porter stemming"]
P["RRF fusion\nk=60 · equal weighting"]
N --> P
O --> P
end
F <-->|"hybrid search"| P
subgraph SOUL ["utils/personality.py"]
Q["soul.md\nSHA-256 drift detection"]
R["SQLite / Postgres\nversion history · TTL prune"]
Q <--> R
end
H <-->|"soul preamble\n≤ 8000 chars"| Q
K <-->|"soul preamble"| Q
subgraph OOB ["Out-of-band — never imported by gate/graph/MCP"]
S["agentic/cli.py\nGitHub read ops"]
T["agentic/fsconnect/\nscoped FS read/write"]
U["sync/cli.py\nDropbox corpus pull"]
V["guardrails/\nNeMo rails skeleton"]
end
style GATEWAY fill:#1a3a5c,color:#ffffff,stroke:#4a90d9
style GRAPH fill:#1a3a2a,color:#ffffff,stroke:#4a9d5a
style RETRIEVAL fill:#3a2a1a,color:#ffffff,stroke:#d9904a
style SOUL fill:#3a1a3a,color:#ffffff,stroke:#d94ad9
style OOB fill:#2a2a2a,color:#aaaaaa,stroke:#666666,stroke-dasharray:5 5
style J fill:#5c1a1a,color:#ffffff
style W fill:#5c1a1a,color:#ffffff
style L fill:#1a1a3a,color:#ffffff
CyClaw's soul mutation endpoints (/soul/propose, /soul/apply, /soul/reload, /soul/restore) require a Bearer API key. Without it they return HTTP 401 immediately — intentional fail-closed behavior.
All
/soul/*endpoints — includingGET /soul— require a validAuthorization: Bearer <key>token. Only/healthand/queryare unauthenticated.
Set for the current session only (cleared on terminal close):
$env:CYCLAW_API_KEY = "your-strong-local-secret"
uvicorn gate:app --host 127.0.0.1 --port 8787Persist across sessions (writes to the current user's environment permanently):
[System.Environment]::SetEnvironmentVariable(
"CYCLAW_API_KEY",
"your-strong-local-secret",
[System.EnvironmentVariableTarget]::User
)
# Restart your terminal, then launch normally:
uvicorn gate:app --host 127.0.0.1 --port 8787Verify it is set before launching:
echo $env:CYCLAW_API_KEYset CYCLAW_API_KEY=your-strong-local-secret
uvicorn gate:app --host 127.0.0.1 --port 8787Persist permanently (takes effect in new sessions):
setx CYCLAW_API_KEY "your-strong-local-secret"[System.Environment]::SetEnvironmentVariable(
"CYCLAW_API_KEY",
"your-strong-local-secret",
[System.EnvironmentVariableTarget]::Machine
)Or via GUI: System Properties → Advanced → Environment Variables → System variables → New.
Set for the current session:
export CYCLAW_API_KEY="your-strong-local-secret"
uvicorn gate:app --host 127.0.0.1 --port 8787Persist in your shell profile (~/.bashrc, ~/.zshrc, or ~/.profile):
echo 'export CYCLAW_API_KEY="your-strong-local-secret"' >> ~/.bashrc
source ~/.bashrc
uvicorn gate:app --host 127.0.0.1 --port 8787Persist for a systemd service (Linux server):
# /etc/systemd/system/cyclaw.service
[Unit]
Description=CyClaw RAG Gateway
After=network.target
[Service]
Type=simple
User=cyclaw
WorkingDirectory=/opt/CyClaw
Environment="CYCLAW_API_KEY=your-strong-local-secret"
Environment="GROK_API_KEY=offline-dummy-sk-123"
ExecStart=/opt/CyClaw/.venv/bin/uvicorn gate:app --host 127.0.0.1 --port 8787
Restart=on-failure
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now cyclawCreate .env in the repo root:
API KEYS added to config.yaml
CYCLAW_API_KEY=api-key
GROK_API_KEY=InputAPIkey
CLAUDE_API_KEY=InputAPIkey
Load it before launching:
# Bash / Zsh
export $(grep -v '^#' .env | xargs)
uvicorn gate:app --host 127.0.0.1 --port 8787# PowerShell
Get-Content .env | ForEach-Object {
if ($_ -match '^([^#=][^=]*)=(.*)$') {
[System.Environment]::SetEnvironmentVariable($Matches[1].Trim(), $Matches[2].Trim())
}
}
uvicorn gate:app --host 127.0.0.1 --port 8787CyClaw is loopback-only (127.0.0.1:8787) — the key never crosses a network. Still:
- Use at least 20 random characters:
openssl rand -hex 20(Linux/macOS) or[System.Web.Security.Membership]::GeneratePassword(24,4)(PowerShell) - Do not reuse a password from elsewhere
- Do not commit the key to Git (
.envis already in.gitignore)
| Requirement | Version | Notes |
|---|---|---|
| Python | 3.12 | Primary supported runtime |
| LM Studio | Any | Must be running on localhost:1234 |
| GGUF model loaded in LM Studio | — | mistral-7b-instruct or qwen2.5-7b work well |
git clone https://github.com/CGFixIT/CyClaw
cd CyClaw
python3.12 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# 1) Install CPU-only torch first (CVE-2025-32434 fixed in 2.6.0; 2.12.1 is within the patched range)
pip install torch==2.12.1+cpu --index-url https://download.pytorch.org/whl/cpu
# 2) Install the rest, pinned to the verified transitive tree.
pip install -r requirements.txt -c constraints.txtmkdir -p data/personality index logs
printf '# Soul\n' > data/personality/soul.md
export GROK_API_KEY=dummypython -m retrieval.indexer
uvicorn gate:app --host 127.0.0.1 --port 8787Open / for the terminal UI and /health for readiness. The terminal exposes three operator consoles — Soul, Sync, and Agentic — the latter two calling POST /ops/sync and POST /ops/agentic (API-key gated, rate-limited, audited).
CyClaw/
├── gate.py
├── graph.py
├── config.yaml # single source of truth
├── README.md
├── Dropbox_Sync_Guide.md
├── mcp_hybrid_server.py # retrieval-only MCP server
├── agentic/ # out-of-band GitHub context + governed registry
│ ├── cli.py
│ ├── context.py
│ ├── gh_client.py
│ ├── registry.py
│ ├── writer.py # stubbed write scaffold, non-executing
│ ├── fsconnect/ # (v1.8) local/SMB filesystem connector
│ │ ├── cli.py
│ │ ├── client.py # scoped reads (fs_list/stat/read/grep)
│ │ ├── pathsafe.py # TOCTOU-safe openat/O_NOFOLLOW security core
│ │ ├── writer.py # gated, atomic writes (default-disabled)
│ │ └── indexer.py # toggleable RAG-corpus indexing of the share
│ └── sqlconnect/ # (v1.8) read-only SQL scaffold (Postgres/MSSQL)
│ ├── cli.py
│ └── client.py # SELECT-only query guard, env-only DSN
├── guardrails/ # (v1.8) optional NeMo Guardrails layer (out-of-band)
│ ├── cli.py
│ ├── config.py
│ ├── integration.py # soft-imports nemoguardrails; degrades gracefully
│ ├── rails.py # offline heuristic rails (injection/soul/grounding)
│ ├── metrics.py # separate logs/guardrails.jsonl stream (hashes only)
│ └── config/ # NeMo config.yml + rails.co (Colang flows)
├── .claude/ # local operator workflows and prompts
│ ├── commands/
│ ├── hooks/
│ ├── memory/
│ ├── patterns/
│ ├── rules/
│ ├── skills/
│ ├── tools/
│ └── utility-prompts/
├── retrieval/
│ ├── indexer.py
│ ├── hybrid_search.py
│ ├── embeddings.py
│ └── stemmer.py
├── llm/
│ └── client.py
├── sync/ # optional Dropbox corpus sync
│ ├── cli.py
│ ├── runner.py
│ └── scheduler.py
├── utils/
│ ├── sanitizer.py
│ ├── logger.py
│ ├── personality.py
│ ├── health.py
│ └── ratelimit.py
├── tests/
├── docs/
├── static/
├── data/
│ ├── corpus/
│ └── personality/
└── .github/workflows/
CyClaw includes an optional, out-of-band Dropbox sync layer that mirrors a Dropbox corpus into data/corpus/ without touching gate.py, graph.py, or the MCP request path.
Key capabilities
rclone-backed pull sync with safety fuses (max_delete,max_transfer)- audit logging for changed corpus files
- optional scheduler integration for Linux and Windows
- optional reindex trigger when corpus changes
Core commands
python -m sync.cli test
python -m sync.cli sync --dry-run
python -m sync.cli sync
python -m sync.cli status
python -m sync.cli schedule
python -m sync.cli unscheduleThe same actions are available from the Sync Console panel in the terminal UI via POST /ops/sync (loopback-only, API-key gated, audited).
See Dropbox_Sync_Guide.md for full setup and scheduling details.
CyClaw now includes a concise, governed agentic layer for local operator workflows. It is opt-in, disabled by default, and fully out-of-band: it is never imported by gate.py, graph.py, or mcp_hybrid_server.py.
- Read-only GitHub context through the
ghCLI - Governed local skills registry with explicit human gating
- Project workflows and operator helpers under
.claude/ - Reusable local patterns for memory, commands, tools, hooks, and utility prompts
- reads only in normal operation
- no GitHub token is stored or forwarded by CyClaw
ghis invoked as an argv list, not via shell execution- write behavior remains scaffolded and non-executing in the current release
- all agentic reads, refusals, and registry changes are audit logged
agentic:
enabled: true
repo: "CGFixIT/CyClaw"
mode: "read"
writes_enabled: false
gh_min_version: "2.40.0"
registry_path: "data/agentic/skills_registry.json"python -m agentic.cli status
python -m agentic.cli context --repo
python -m agentic.cli context --pr 123
python -m agentic.cli context --issue 45
python -m agentic.cli test
python -m agentic.cli propose-skill --name deploy --desc "..." --body-file s.md --reason "draft"
python -m agentic.cli apply-skill --name deploy --desc "..." --body-file s.md --reason "add deploy runbook" --confirmThe Agentic Console panel drives these from the terminal UI via POST /ops/agentic; skill-Apply stays disabled behind a 4-gate checklist (mode=write + writes_enabled + reason + --confirm) — dry-run only under shipped defaults.
Key areas in agentic folders
skills/— reusable project skills / workflowscommands/— shortcut command entry pointspatterns/— repeatable operating patternstools/— tool wrappers and helper definitionsutility-prompts/— reusable operator promptsmemory/— memory-oriented helpers / artifactshooks/andrules/— local guardrails and automation boundaries
Examples from the current repo
- run / smoke-test workflows for CyClaw
- architecture, tests, logging, and speed refactor loops
- wrap-up / session-end workflows
- memory orchestration support patterns
v1.8 extends the agentic layer beyond GitHub to local data, for the regulated or security conscious use case where AI use is compliance heavy. Both connectors are opt-in, disabled by default, and fully out-of-band — never imported by gate.py, graph.py, or mcp_hybrid_server.py, so the five security invariants hold by construction. While disabled, their CLIs are a pure no-op (exit 0).
Scoped reads and separately-gated writes over a local or SMB file share, sharing one TOCTOU-safe security core.
pathsafe.pysecurity core — POSIXopenat/O_NOFOLLOWhandle-descent from a held root directory fd (so the root cannot be swapped under the process). Denies UNC, NTFS alternate data streams (file::$DATA),\\?\/\\.\device paths,..traversal, and any symlink / reparse point. Segment-aware containment closes CVE-2025-53110 (sibling-prefix) andrealpath+O_NOFOLLOWclose CVE-2025-53109 (symlink/junction escape).- Reads (
fs_list/fs_stat/fs_read/fs_grep) confined toallowed_roots, audited, with a 5 MiB read cap and advisory OWASP∪banned_patternscontent scanning. - Writes (
fs_write/fs_append/fs_mkdir/fs_move) — fully built butwrites_enabled: falseby default; confined to a separatewritable_rootslist; gated by a humanreason+--confirm(for destructive ops); atomic (tmp+os.replace); content-agnostic (never calls the LLM — an operator pipes local-LLM/QWEN output in). A code-levelFS_WRITE_HARD_DISABLEkill switch forces dry-run regardless of config. - Toggleable RAG-corpus indexing of the share (
index_enabled, dry-run default) stages eligible files into the corpus and triggers a reindex subprocess — enabling a generate → write → index loop without importing the retrieval layer.
python -m agentic.fsconnect.cli status
python -m agentic.fsconnect.cli read <path> # scoped read
python -m agentic.fsconnect.cli grep <path> <pattern>
python -m agentic.fsconnect.cli write <path> --reason "..." # dry-run unless writes_enabled
python -m agentic.fsconnect.cli index --apply # stage share → corpus
python -m agentic.fsconnect.cli test # pre-flight self-testEnable in config.yaml:
fsconnect:
enabled: true
allowed_roots: ["/srv/share"] # REQUIRED when enabled; existing dirs
max_file_bytes: 5242880 # 5 MiB read cap
writes_enabled: false # master write switch (dry-run plans while false)
writable_roots: [null] # null => OS default (/var/lib/cyclaw-fs | C:\CyClaw-FS)
max_write_bytes: 10485760 # 10 MiB write cap
index_enabled: false # toggle RAG-corpus indexing of the shareA disabled-by-default scaffold for read-only on-prem SQL (Postgres / MSSQL). Read-only is enforced three ways: a SELECT/WITH-only query guard (rejects DDL/DML, stacked statements, and comment-hidden keywords by scanning a quote-stripped copy), a session-level read-only transaction, and a hard allow_write: false. The DSN is read from an environment variable only (CYCLAW_SQL_DSN), never hardcoded; drivers (psycopg / pyodbc) are imported lazily.
python -m agentic.sqlconnect.cli status
python -m agentic.sqlconnect.cli schema # list table schemas (read-only)
python -m agentic.sqlconnect.cli query --table public.users # bounded preview
python -m agentic.sqlconnect.cli testsqlconnect:
enabled: false
driver: "postgres" # "postgres" | "mssql"
dsn_env: "CYCLAW_SQL_DSN" # DSN from this env var only
statement_timeout_ms: 5000
max_rows: 1000
allow_write: false # reserved; v0.1 cannot write regardlessAn opt-in content-safety layer in guardrails/. Absence of the guardrails: block, or enabled: false (the shipped default), is a pure no-op. Since Phase 2, when enabled, utils/guardrail_bridge.py wires its offline input rail into one visible graph.py node (guardrail_input, between route_by_score and local_llm) — still defense-in-depth only, never a routing authority: the graph's own guardrail_router edge (topology, not guardrails code) decides where a blocked query goes. guardrails itself is still never imported directly by gate.py or graph.py — utils/guardrail_bridge.py is the only seam, preserving module isolation (invariant I6). mcp_hybrid_server.py never touches it at all.
nemoguardrailsis an optional dependency. The layer soft-imports it and, when it is absent, degrades to offline heuristic rails that need no second LLM call:- input — light prompt-injection marker scan + soul/identity-mutation intent detection (the content-layer arm of the Soul-Governance invariant);
- output — token-overlap grounding check against the retrieved context, flagging likely-ungrounded (hallucinated) answers below
hallucination_threshold.
- When
nemoguardrailsis installed, the same Python checks back the live NeMo actions (via the Colang flows inguardrails/config/rails.co), so the offline heuristics and live rails never drift. - Decisions are recorded to a separate metrics stream (
logs/guardrails.jsonl) that stores only SHA-256 hashes — never raw text — mirroring the audit log's privacy posture.
python -m guardrails.cli status # config + nemoguardrails availability
python -m guardrails.cli check "your query here" # run offline rails (no LLM/NeMo needed)
python -m guardrails.cli metrics # summarize the guardrail stream
python -m guardrails.cli test # pre-flight self-testguardrails:
enabled: false # opt-in; also gates the graph.py guardrail_input node
engine: "openai" # LM Studio / Ollama OpenAI-compatible endpoint
model: "qwen2.5-7b-instruct" # keep in sync with models.local_llm.model
hallucination_threshold: 0.18 # token-overlap floor for the grounding rail
metrics_path: "logs/guardrails.jsonl" # separate from logs/audit.jsonl (hashes only)Full design / wiring plan:
docs/NeMo/later_development_guideline.md. Phase 2 implementation contract:docs/NeMo/phase2_implementation_plan.md.
For Claude Desktop or other MCP-compatible clients:
{
"mcpServers": {
"cyclaw": {
"command": "python",
"args": ["/path/to/CyClaw/mcp_hybrid_server.py"]
}
}
}The MCP server exposes a retrieval-only hybrid_search tool. It has no sampling capability and is intentionally isolated from the agentic, filesystem/SQL connector, NeMo Guardrails, and Dropbox corpus sync layers.
| Layer | Mechanism |
|---|---|
| Network | Binds 127.0.0.1:8787 — no external exposure by design |
| Input | Config-driven injection filter (policy.prompt_filter) |
| Rate limit | 60 req/min per IP |
| Telemetry | Kill block runs before any SDK import in gate.py |
| Audit | All paths log SHA-256 query hash + PII-redacted metadata |
| Grok gating | Triple gate: mode=hybrid AND grok.enabled=true AND user_confirmed_online=true |
| Claude gating | Same triple gate, independently: mode=hybrid AND claude.enabled=true AND user_confirmed_online=true |
| Soul writes | Explicit human reason string + enforced write-boundary scan + atomic write |
| Agentic writes | Stubbed / non-executing in current release |
| Filesystem connector | Reads scoped to allowed_roots (5 MiB cap); writes default-OFF, confined to a separate writable_roots, gated by human reason + --confirm, atomic; TOCTOU-safe pathsafe core denies UNC/ADS/device-path/../symlink escapes |
| SQL connector | Read-only: SELECT/WITH-only query guard + session read-only + hard allow_write: false; DSN from env var only; disabled scaffold by default |
| Guardrails | Out-of-band, opt-in defense-in-depth; degrades to offline heuristic rails without nemoguardrails; never a routing authority; separate hash-only metrics stream |
/ops/* routes |
Loopback-only, require_api_key gated, rate-limited (60/min), every call audited (ops_sync_executed / ops_agentic_executed); shells out via subprocess.run([...]) — never imports sync/ or agentic/ |
| Container | Non-root, no-new-privileges, cap_drop: ALL, read-only rootfs, seccomp, resource limits; optional eBPF/Falco detection (deploy/falco/, off by default) |
Scope: CyClaw is a single-operator, loopback-bound local server. The full threat model — what the sandbox does and does not cover (no microVM by design) and why — is documented in
docs/THREAT_MODEL.md.
Built by Chris Grady · cgfixit.com/linkedin