Feature: Neural Mesh RAG — Self-Evolving Knowledge Architecture

[mrveiss]

Summary

Transform AutoBot's retrieval system from a fixed pipeline into a self-evolving knowledge mesh. Minimal bootstrap seed, autonomous evolution through Hebbian reinforcement ("nodes that fire together, wire together").

Architecture

Two Layers:

• Seed Layer (built once): Chunk nodes + embeddings (ChromaDB), structural edges (PART_OF, NEXT, SIMILAR_TO), entity-based edges
• Mesh Brain (AutoBot manages): EdgeLearner, EdgeDiscoverer, MeshPruner, NodePromoter

Three Storage Layers:

• ChromaDB: vectors + anchor summaries
• Redis: hot edges for retrieval-time expansion
• PostgreSQL: mesh adjacency tables + evolution log (Mesh Brain workspace)

Implementation Phases

Phase 1: Harden Foundations ✅

• Wire publish_live_event() producers (feat: wire publish_live_event() into agent/task/workflow services (#1408 follow-up) #1516 → PR feat(rag): emit retrieval feedback events (#1516) #2023)
• BM25 keyword search (Improvement: Keyword search uses TF-only scoring — upgrade to BM25 for better retrieval #1720 → PR feat(search): upgrade keyword search to BM25 scoring (#1720) #2021)
• Query complexity classifier (Feature: Dynamic per-query hybrid search weight selection #1719 → PR feat(search): add query complexity classifier (#1719) #2018)
• Configurable reranker blend (feat: Configurable reranker blend weights (Neural Mesh RAG Phase 1) #2004 → PR feat(search): configurable reranker blend weights (#2004) #2020, Bug: RAGConfig.rerank_weights not forwarded to reranker #2034 → PR fix(rag): forward rerank_weights from config to reranker (#2034) #2075)
• RAG retrieval feedback hook (feat: Feedback hook integration with complexity classifier (Neural Mesh RAG Phase 1) #2024 → PR feat(rag): integrate complexity classifier into feedback hook (#2024) #2038)
• Context tracker (feat: Context tracker for multi-step retrieval (Neural Mesh RAG Phase 1) #2005 → PR feat(search): add context tracker for multi-step retrieval (#2005) #2019)

Phase 2: ECL + Dual Indexing ✅

• Dual-mode entity extraction (feat: Dual-mode entity extraction — LLM + NLP (Neural Mesh RAG Phase 2) #2025 → PR feat(pipeline): add NLP-light entity extraction (#2025) #2036)
• Relationship extraction (feat: Dual-mode relationship extraction — LLM + NLP (Neural Mesh RAG Phase 2) #2026 → PR feat(pipeline): add NLP-light relationship extraction (#2026) #2042)
• RAPTOR recursive summarizer (feat: RAPTOR recursive summarizer (Neural Mesh RAG Phase 2) #2027 → PR feat(pipeline): add RAPTOR recursive clustering summarizer (#2027) #2040, feat: Wire RAPTOR build_raptor_tree into summarizer process() pipeline #2051 → PR feat(pipeline): wire RAPTOR build_raptor_tree into process() (#2051) #2074)
• MeshSeeder loader (feat: MeshSeeder loader for graph edge creation (Neural Mesh RAG Phase 2) #2028 → PR feat(pipeline): add MeshSeeder loader (#2028) #2041, feat: MeshSeeder missing SIMILAR_TO cosine edge builder (#2028 follow-up) #2049 → PR feat(pipeline): add SIMILAR_TO cosine edges to MeshSeeder (#2049) #2065, Bug: MeshSeeder uses synthetic node IDs incompatible with Phase 3 PostgreSQL schema #2050 → PR fix(pipeline): use chunk_id as canonical node ID in MeshSeeder (#2050) #2062)
• PostgreSQL → Redis edge sync (feat: PostgreSQL to Redis edge sync service (Neural Mesh RAG Phase 2) #2029 → PR feat(mesh): add edge sync service (#2029) #2039)

Phase 3: Mesh Brain + NeuralMeshRetriever ✅

• PostgreSQL mesh schema migration (feat: PostgreSQL mesh schema migration (Neural Mesh RAG Phase 3) #2055 → PR feat(mesh): add PostgreSQL mesh schema and async MeshDB client (#2055) #2108)
• EdgeLearner (feat: EdgeLearner — Hebbian reinforcement from retrieval feedback (Neural Mesh RAG Phase 3) #2056 → PR feat(mesh): add EdgeLearner — Hebbian reinforcement (#2056) #2099)
• PersonalizedPageRank (feat: PersonalizedPageRank for mesh graph expansion (Neural Mesh RAG Phase 3) #2057 → PR feat(mesh): add PersonalizedPageRank for graph expansion (#2057) #2104)
• NeuralMeshRetriever (feat: NeuralMeshRetriever — unified mesh-aware retrieval (Neural Mesh RAG Phase 3) #2058 → PR feat(mesh): add NeuralMeshRetriever — unified mesh-aware retrieval (#2058) #2114)
• Feature flag integration (feat: RAGService mesh_retriever integration behind feature flag (Neural Mesh RAG Phase 3) #2059 → PR feat(rag): add mesh feature flags and RAGService integration (#2059) #2112)

Phase 4: Autonomous Evolution ✅

• EdgeDiscoverer (feat: EdgeDiscoverer — nightly LLM-based relationship naming (Neural Mesh RAG Phase 4) #2117 → PR feat(mesh): add EdgeDiscoverer — LLM relationship naming (#2117) #2129)
• MeshPruner (feat: MeshPruner — weekly entropy control and decay (Neural Mesh RAG Phase 4) #2118 → PR feat(mesh): add MeshPruner — weekly entropy control (#2118) #2130)
• NodePromoter (feat: NodePromoter — daily anchor emergence (Neural Mesh RAG Phase 4) #2119 → PR feat(mesh): add NodePromoter — daily anchor emergence (#2119) #2131)
• Mesh Brain scheduler (feat: MeshBrainScheduler — orchestrate background mesh jobs (Neural Mesh RAG Phase 4) #2120 → PR feat(mesh): add MeshBrainScheduler + health API (#2120) #2132)
• Runtime feature flag progression (included in feat: RAGService mesh_retriever integration behind feature flag (Neural Mesh RAG Phase 3) #2059)

Phase 5: Agentic RAG + Mesh Agent Topology ✅

• Query decomposer (feat: Query decomposer — MA-RAG pattern for MULTI_HOP queries (Neural Mesh RAG Phase 5) #2134 → PR feat(mesh): add QueryDecomposer — MA-RAG multi-hop (#2134) #2147)
• Evidence extractor (feat: Evidence extractor — sentence-level precision (Neural Mesh RAG Phase 5) #2135 → PR feat(mesh): add EvidenceExtractor — sentence-level precision (#2135) #2150)
• Autonomous strategy selection (feat: Autonomous strategy selection — A-RAG ReAct loop (Neural Mesh RAG Phase 5) #2136 → PR feat(mesh): add A-RAG autonomous strategy selection (#2136) #2156)
• Agent mesh topology (feat: Agent mesh topology — dynamic DAG with Hebbian evolution (Neural Mesh RAG Phase 5) #2137 → PR feat(mesh): add AgentTopology — dynamic DAG (#2137) #2149)
• Topology-aware routing + agent specialization (feat: Topology-aware routing + agent specialization (Neural Mesh RAG Phase 5) #2138 → PR feat(mesh): topology-aware routing + agent specialization (#2138) #2151)
• LangChain 1.x migration (security: migrate langchain-core from 0.3.x to 1.2.11+ (CVE fix) #1572 — deferred, not blocking)

Final Stats

• 33 PRs merged
• ~310 tests
• 5 phases completed in 2 sessions

Absorbs Issues

• Feature: Agentic RAG — expose search as LLM tool with query rewriting and iterative retrieval #1718 Agentic RAG ✅
• Feature: Dynamic per-query hybrid search weight selection #1719 Dynamic per-query hybrid weights ✅
• Improvement: Keyword search uses TF-only scoring — upgrade to BM25 for better retrieval #1720 BM25 keyword search ✅

Prerequisites

• feat: wire publish_live_event() into agent/task/workflow services (#1408 follow-up) #1516 Wire publish_live_event() ✅
• security: migrate langchain-core from 0.3.x to 1.2.11+ (CVE fix) #1572 LangChain 1.x migration (deferred — not blocking Phase 5 core features)
• feat: Full-stack dependency upgrade — Python 3.12 + latest pip/npm packages #1836 Dependency upgrade — spaCy + scikit-learn added (Bug: Missing spacy and scikit-learn in requirements.txt #2043 ✅)

Design Document

docs/plans/2026-03-22-neural-mesh-rag-design.md

Key Research Sources

AgentNet (NeurIPS 2025), G-Designer, MA-RAG, A-RAG, GraphRAG (Microsoft), HippoRAG (NeurIPS 2024), RAPTOR (ICLR 2024), LightRAG, LazyGraphRAG (Microsoft), Modular RAG