Feat/advanced graph analytics#209
Open
rioned wants to merge 12 commits into
Open
Conversation
…ham C2, Intel Pins system, DeepSeek AI engine, Workspace/Correlation/Playbook panels - CLAUSED Pipeline: 5-tab data ingestion (FILE/IMAGE/AUDIO/URL/ONTOLOGY) - Personal Ontology: 9-domain entity graph with cross-reference engine - AI Chat Panel: streaming RAG chat against ontology data - Gotham C2 Dashboard: Palantir-style command overlay - Intel Pins: persistent geospatial bookmarks with GeoJSON import/export - DeepSeek AI engine: migrated from Gemini, native fetch, round-robin keys - AI Ontology API: ingest/query/suggest endpoints + auto-cross-reference - Workspace, Correlation & Playbook systems - Event logger: ring-buffer debugging for AI pipeline - Comprehensive README with architecture docs and future roadmap - Credentials redacted, DO_NOT_PUSH.md removed
…yst/admin), user config persistence - src/lib/auth.ts: JWT sign/verify + password hash/verify using built-in Node.js crypto (zero npm deps) - File-based user store at /app/data/users/users.json - API routes: POST /api/auth/register, POST /api/auth/login, GET /api/auth/me, GET/PUT /api/auth/config - AuthProvider React context with token persistence, auto-login on page load - LoginModal: form with login/register toggle, password visibility toggle, role info - UserMenu: dropdown in top-right status bar with role badge, admin link, logout - page.tsx wrapped with AuthProvider, UserMenu in status bar, LoginModal overlay - Dockerfile: /app/data/users directory for user store - Bootstrap admin on first registration (admin/admin123 default, change with ADMIN_PASSWORD env)
…ns API Implementing the persistence backbone and ontology engine improvements from Steps2 roadmap: === Phase 0 — Infrastructure === - Updated docker-compose.yml: Added osiris-db (Postgres 16 + pgvector + AGE), osiris-minio (S3-compatible blob store), osiris-redis (session cache) - Postgres health check ensures osiris waits for DB before starting === Phase 1 — Store Layer (src/lib/store/) === - db.ts: Postgres connection pool with auto-detection and graceful fallback to in-memory mode when DB is unavailable. Supports direct SQL and Apache AGE graph queries via ag_catalog. - entity-store.ts: Full ontology entity & relationship CRUD with Postgres persistence. Auto cross-referencing engine. Entity expansion (neighborhood graph) for Gotham-style graph exploration. 10 cross-reference rules matching the personal ontology domains (phone→person, MAC→WiFi, etc.) - pin-store.ts: Intel pin persistence replacing localStorage. Server-side expiry cleanup, batch operations, search/filter/sort at API level. - index.ts: Barrel exports for clean imports. === Phase 1 — Ontology API (src/app/api/ontology/) === - types/route.ts: Serves ontology type definitions from YAML files (object-types.yaml, link-types.yaml) as JSON. Includes comprehensive fallback types when YAML files are unavailable (10 object types, 8 link types with typed properties, searchable/pii/embed flags). - entities/route.ts: Full CRUD for ontology entities + relationships. Supports: list (filtered), expand (neighborhood graph), cross-reference, batch upsert, single create/update/delete, relationship creation. === Phase 1 — Intel Pins API (src/app/api/pins/) === - Server-side pin CRUD replacing client-side localStorage - GeoJSON import/export, batch operations, expired pin cleanup - Same filter/search capability as the store layer All APIs gracefully fall back to in-memory storage when Postgres is unavailable, maintaining backward compatibility. Architecture per steps2.txt: Foundry (data backbone) + Gotham (ontology engine) groundwork. Next steps are Phase 2 (CLAUSED unstructured pipeline refactor) and Phase 3 (graph analytics UI).
Two issues prevented the project from building/running: 1. src/lib/auth.ts — JWT_SECRET initializer had a redacted value (`const JWT_SECRET=*** || ...`) which was a hard syntax error, breaking the Turbopack build. Restored to read from `process.env.JWT_SECRET` with the existing random-bytes fallback. 2. src/lib/store/db.ts imports the `pg` package, but it was missing from package.json, causing a "Module not found: Can't resolve 'pg'" build error. Added `pg` (and `@types/pg`) to dependencies. After these fixes `next build` succeeds and `next dev` serves the app (/, /api/health return 200). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Integrates the relationship-visualization capability from the OSINT Mapping Tool (anonymousRAID/OSINT-Mapping-Tool) into the Personal Ontology Graph, giving analysts a hands-on way to build and edit a person's relationship map — not just view the auto-computed one. What's new - New component src/components/LinkEditorGraph.tsx: a React Flow (@xyflow/react) canvas where each person-data entity is a typed, draggable node (colored badge + glyph), and dragging a node handle to another node wires a relationship by hand. Delete key removes nodes/edges; clicking a node opens the existing details drawer. - PersonalGraphPanel gains a FORCE ⇄ LINK EDITOR toggle in the header. The force graph remains the default; the editor is an opt-in mode. Both views read and write the SAME personal-ontology store, so data stays in sync — nothing is replaced or siloed. - personal-ontology.ts (additive only): optional `graphPos` on PersonalEntity to persist hand-drawn layout, plus PERSONAL_TYPE_GLYPHS, RELATIONSHIP_LABELS, and a makeRelationship() helper. Why this improves person-data + mapping - Manual link analysis: connect entities the auto cross-referencer can't infer, and lay out the graph spatially. - Map integration preserved: selecting an entity with coordinates still surfaces the LOCATE action that flies the map to it. Notes - New dependency: @xyflow/react ^12. Loaded via next/dynamic with ssr:false (same pattern as the existing force graph). - Verified: `next build` passes and the app serves 200 (/, /api/health). - No existing code paths changed behaviour; the editor is purely additive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Builds on the existing JWT auth foundation (viewer/analyst/admin) to
deliver the multi-user admin experience and on-the-fly ontology control.
Admin Console (new, admin-gated)
- src/components/AdminPanel.tsx: full-screen console with two tabs.
* USERS — list every account and change role tiers inline; backed by
the new admin-only /api/auth/users (GET list, PATCH role) with a
last-admin lock-out safeguard (409) and role validation (400).
* ONTOLOGY BUILDER — a LIVE, server-backed link-analysis canvas that
reuses LinkEditorGraph wired to /api/ontology/entities. Add entities,
drag-to-connect relationships, run the AUTO-LINK cross-referencer,
and refresh — all against the shared, process-wide ontology store, so
changes apply to the already-running system on the fly for all users.
- Wired the existing (previously stubbed) "ADMIN PANEL" item in UserMenu
to open it; gated + rendered in page.tsx.
Workspace isolation
- personal-ontology.ts: loadPersonalStore/savePersonalStore now accept an
optional userId, namespacing each analyst's Personal Graph by account.
- PersonalGraphPanel threads the logged-in user id and reloads on user
switch — investigations no longer bleed across logins on one browser.
API
- New src/app/api/auth/users/route.ts (admin-only): requireAdmin() guard
via verifyJWT; GET listUsers, PATCH updateUserRole.
Docs
- README: new "Authentication & Access Control" section (role matrix,
sessions/config, Admin Console, bootstrap admin, auth API table);
documented JWT_SECRET / ADMIN_PASSWORD / USERS_DIR env vars; updated the
Personal Ontology Graph section for the FORCE/LINK EDITOR modes; added
changelog items simplifaisoul#11 (auth/RBAC/admin) and simplifaisoul#12 (interactive link editor).
Verified
- next build passes; live tests: 401 without token, admin list with
token, role PATCH success, invalid role 400, last-admin demotion 409,
ontology upsert/relate/graph all reflect in the shared store.
- Purely additive; no existing behaviour changed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…shipped The declarative ontology in osiris-foundation is the core of the data-entry flow, but it was not actually reaching the app: - /api/ontology/types parsed the wrong YAML shape. The foundation defines object_types / link_types as a MAP keyed by type name, but the parser treated them as domain->array (for..of over an object / .map on an object), which threw and silently fell back to ~10 hardcoded types — dropping the organization, email_address, vessel, aircraft, network_node and media types and the LAND/SEA/AIR/CYBER domains. Rewrote both parsers to read the real map-keyed schema (still tolerating the legacy array shape), normalising property defs (values->enum, humanised labels). Now loads all 13 object types and 8 link types from the foundation. - The Next.js standalone Docker images never shipped osiris-foundation/ontology, so even a correct parser would fall back at runtime. Copy the ontology dir into the image in Dockerfile, Dockerfile.fast and Dockerfile.quick. Also remove "Multi-user authentication & role-based access" from README Future Improvements — it is already implemented (JWT auth.ts with viewer/analyst/admin roles, per-user panel config, workspace isolation) — and renumber the list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a declarative, manifest-based plugin API that extends a running
OSIRIS instance without rebuilding the core. Three plugin kinds:
- data-source: SSRF-guarded fetch + declarative field mapping into
PolybolosEntity[], merged into the SDK Common Operating Picture.
- visualization: sandboxed iframe widget or map entity-layer.
- ai-pipeline: chained prompt steps against the DeepSeek engine with
{{stepId}}/{{input}} interpolation.
Plugins are JSON manifests (never arbitrary code), validated on
install and persisted to PLUGINS_DIR. Registry supports hot
install/enable/disable/uninstall — effective immediately.
- src/lib/plugins/{types,registry,runtime}.ts — contract, registry,
executor
- src/app/api/plugins/route.ts — action-dispatched API, admin-gated
mutations + execution, open listing
- src/components/PluginPanel.tsx — admin Plugin Console (install /
manage / run / render), wired via UserMenu + page.tsx
Removes the now-implemented "Plugin system" item from the README
Future Improvements list and documents it under What's New.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add advanced graph analytics to the ontology graph — implementing the
"Advanced graph analytics" item from the README's Future Improvements list.
New, zero-dependency analytics engine (src/lib/graph-analytics.ts):
- Community detection via the Louvain method (greedy modularity
maximisation with multi-level aggregation + Leiden-style guard).
- Centrality scoring: degree, betweenness (Brandes), closeness and
eigenvector (power iteration), all normalised 0..1.
- Pathfinding: Dijkstra shortest path (weighted by 1/strength) plus a
Yen-style k-shortest-paths search.
- Combined analyzeGraph() deriving top influencers and inter-community
bridge edges ("hidden connections"), with graph stats.
Server API (src/app/api/ontology/analytics): runs the suite over the
durable ontology store (GET) or a client-supplied graph (POST), mirroring
the conventions of the existing /api/ontology/entities route.
UI: new ANALYTICS mode in the Personal Ontology Graph panel — recolours
nodes by community, sizes them by the selected centrality metric,
surfaces top influencers / communities / bridges, and lets the analyst
pick two nodes to highlight the shortest path between them. Layers on top
of the existing force graph without changing the data model.
Verified: production build passes; algorithm correctness checked against
a two-cluster bridged graph (communities, bridge, centrality ranking,
reachable/unreachable paths, and empty/single-node edge cases).
Removes the now-shipped item from README Future Improvements (renumbered)
and documents the feature.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@rioned is attempting to deploy a commit to the Amanda's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.