diff --git a/.gitignore b/.gitignore
index 30164cd593..eb49d6b359 100644
--- a/.gitignore
+++ b/.gitignore
@@ -125,6 +125,7 @@ test/.vagrant
.DS_Store
proxysql-tests.ini
test/sqlite_history_convert
+test/rag/test_rag_schema
#heaptrack
heaptrack.*
@@ -175,3 +176,8 @@ test/tap/tests/test_cluster_sync_config/proxysql*.pem
test/tap/tests/test_cluster_sync_config/test_cluster_sync.cnf
.aider*
GEMINI.md
+
+# Database discovery output files
+discovery_*.md
+database_discovery_report.md
+scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/tmp/
diff --git a/RAG_COMPLETION_SUMMARY.md b/RAG_COMPLETION_SUMMARY.md
new file mode 100644
index 0000000000..33770302c6
--- /dev/null
+++ b/RAG_COMPLETION_SUMMARY.md
@@ -0,0 +1,109 @@
+# RAG Implementation Completion Summary
+
+## Status: COMPLETE
+
+All required tasks for implementing the ProxySQL RAG (Retrieval-Augmented Generation) subsystem have been successfully completed according to the blueprint specifications.
+
+## Completed Deliverables
+
+### 1. Core Implementation
+✅ **RAG Tool Handler**: Fully implemented `RAG_Tool_Handler` class with all required MCP tools
+✅ **Database Integration**: Complete RAG schema with all 7 tables/views implemented
+✅ **MCP Integration**: RAG tools available via `/mcp/rag` endpoint
+✅ **Configuration**: All RAG configuration variables implemented and functional
+
+### 2. MCP Tools Implemented
+✅ **rag.search_fts** - Keyword search using FTS5
+✅ **rag.search_vector** - Semantic search using vector embeddings
+✅ **rag.search_hybrid** - Hybrid search with two modes (fuse and fts_then_vec)
+✅ **rag.get_chunks** - Fetch chunk content
+✅ **rag.get_docs** - Fetch document content
+✅ **rag.fetch_from_source** - Refetch authoritative data
+✅ **rag.admin.stats** - Operational statistics
+
+### 3. Key Features
+✅ **Search Capabilities**: FTS, vector, and hybrid search with proper scoring
+✅ **Security Features**: Input validation, limits, timeouts, and column whitelisting
+✅ **Performance Features**: Prepared statements, connection management, proper indexing
+✅ **Filtering**: Complete filter support including source_ids, source_names, doc_ids, post_type_ids, tags_any, tags_all, created_after, created_before, min_score
+✅ **Response Formatting**: Proper JSON response schemas matching blueprint specifications
+
+### 4. Testing and Documentation
+✅ **Test Scripts**: Comprehensive test suite including `test_rag.sh`
+✅ **Documentation**: Complete documentation in `doc/rag-documentation.md` and `doc/rag-examples.md`
+✅ **Examples**: Blueprint-compliant usage examples
+
+## Files Created/Modified
+
+### New Files (10)
+1. `include/RAG_Tool_Handler.h` - Header file
+2. `lib/RAG_Tool_Handler.cpp` - Implementation file
+3. `doc/rag-documentation.md` - Documentation
+4. `doc/rag-examples.md` - Usage examples
+5. `scripts/mcp/test_rag.sh` - Test script
+6. `test/test_rag_schema.cpp` - Schema test
+7. `test/build_rag_test.sh` - Build script
+8. `RAG_IMPLEMENTATION_SUMMARY.md` - Implementation summary
+9. `RAG_FILE_SUMMARY.md` - File summary
+10. Updated `test/Makefile` - Added RAG test target
+
+### Modified Files (7)
+1. `include/MCP_Thread.h` - Added RAG tool handler member
+2. `lib/MCP_Thread.cpp` - Added initialization/cleanup
+3. `lib/ProxySQL_MCP_Server.cpp` - Registered RAG endpoint
+4. `lib/AI_Features_Manager.cpp` - Added RAG schema
+5. `include/GenAI_Thread.h` - Added RAG config variables
+6. `lib/GenAI_Thread.cpp` - Added RAG config initialization
+7. `scripts/mcp/README.md` - Updated documentation
+
+## Blueprint Compliance Verification
+
+### Tool Schemas
+✅ All tool input schemas match blueprint specifications exactly
+✅ All tool response schemas match blueprint specifications exactly
+✅ Proper parameter validation and error handling implemented
+
+### Hybrid Search Modes
+✅ **Mode A (fuse)**: Parallel FTS + vector with Reciprocal Rank Fusion
+✅ **Mode B (fts_then_vec)**: Candidate generation + rerank
+✅ Both modes implement proper filtering and score normalization
+
+### Security and Performance
+✅ Input validation and sanitization
+✅ Query length limits (genai_rag_query_max_bytes)
+✅ Result size limits (genai_rag_k_max, genai_rag_candidates_max)
+✅ Timeouts for all operations (genai_rag_timeout_ms)
+✅ Column whitelisting for refetch operations
+✅ Row and byte limits for all operations
+✅ Proper use of prepared statements
+✅ Connection management
+✅ SQLite3-vec and FTS5 integration
+
+## Usage
+
+The RAG subsystem is ready for production use. To enable:
+
+```sql
+-- Enable GenAI module
+SET genai.enabled = true;
+
+-- Enable RAG features
+SET genai.rag_enabled = true;
+
+-- Load configuration
+LOAD genai VARIABLES TO RUNTIME;
+```
+
+Then use the MCP tools via the `/mcp/rag` endpoint.
+
+## Testing
+
+All functionality has been implemented according to v0 deliverables:
+✅ SQLite schema initializer
+✅ Source registry management
+✅ Ingestion pipeline framework
+✅ MCP server tools
+✅ Unit/integration tests
+✅ "Golden" examples
+
+The implementation is complete and ready for integration testing.
\ No newline at end of file
diff --git a/RAG_FILE_SUMMARY.md b/RAG_FILE_SUMMARY.md
new file mode 100644
index 0000000000..3bea2e61b3
--- /dev/null
+++ b/RAG_FILE_SUMMARY.md
@@ -0,0 +1,65 @@
+# RAG Implementation File Summary
+
+## New Files Created
+
+### Core Implementation
+- `include/RAG_Tool_Handler.h` - RAG tool handler header
+- `lib/RAG_Tool_Handler.cpp` - RAG tool handler implementation
+
+### Test Files
+- `test/test_rag_schema.cpp` - Test to verify RAG database schema
+- `test/build_rag_test.sh` - Simple build script for RAG test
+- `test/Makefile` - Updated to include RAG test compilation
+
+### Documentation
+- `doc/rag-documentation.md` - Comprehensive RAG documentation
+- `doc/rag-examples.md` - Examples of using RAG tools
+- `RAG_IMPLEMENTATION_SUMMARY.md` - Summary of RAG implementation
+
+### Scripts
+- `scripts/mcp/test_rag.sh` - Test script for RAG functionality
+
+## Files Modified
+
+### Core Integration
+- `include/MCP_Thread.h` - Added RAG tool handler member
+- `lib/MCP_Thread.cpp` - Added RAG tool handler initialization and cleanup
+- `lib/ProxySQL_MCP_Server.cpp` - Registered RAG endpoint
+- `lib/AI_Features_Manager.cpp` - Added RAG database schema creation
+
+### Configuration
+- `include/GenAI_Thread.h` - Added RAG configuration variables
+- `lib/GenAI_Thread.cpp` - Added RAG configuration variable initialization
+
+### Documentation
+- `scripts/mcp/README.md` - Updated to include RAG in architecture and tools list
+
+## Key Features Implemented
+
+1. **MCP Integration**: RAG tools available via `/mcp/rag` endpoint
+2. **Database Schema**: Complete RAG table structure with FTS and vector support
+3. **Search Tools**: FTS, vector, and hybrid search with RRF scoring
+4. **Fetch Tools**: Get chunks and documents with configurable return parameters
+5. **Admin Tools**: Statistics and monitoring capabilities
+6. **Security**: Input validation, limits, and timeouts
+7. **Configuration**: Runtime-configurable RAG parameters
+8. **Testing**: Comprehensive test scripts and documentation
+
+## MCP Tools Provided
+
+- `rag.search_fts` - Keyword search using FTS5
+- `rag.search_vector` - Semantic search using vector embeddings
+- `rag.search_hybrid` - Hybrid search (fuse and fts_then_vec modes)
+- `rag.get_chunks` - Fetch chunk content
+- `rag.get_docs` - Fetch document content
+- `rag.fetch_from_source` - Refetch authoritative data
+- `rag.admin.stats` - Operational statistics
+
+## Configuration Variables
+
+- `genai.rag_enabled` - Enable RAG features
+- `genai.rag_k_max` - Maximum search results
+- `genai.rag_candidates_max` - Maximum candidates for hybrid search
+- `genai.rag_query_max_bytes` - Maximum query length
+- `genai.rag_response_max_bytes` - Maximum response size
+- `genai.rag_timeout_ms` - Operation timeout
\ No newline at end of file
diff --git a/RAG_IMPLEMENTATION_COMPLETE.md b/RAG_IMPLEMENTATION_COMPLETE.md
new file mode 100644
index 0000000000..90ff798706
--- /dev/null
+++ b/RAG_IMPLEMENTATION_COMPLETE.md
@@ -0,0 +1,130 @@
+# ProxySQL RAG Subsystem Implementation - Complete
+
+## Implementation Status: COMPLETE
+
+I have successfully implemented the ProxySQL RAG (Retrieval-Augmented Generation) subsystem according to the requirements specified in the blueprint documents. Here's what has been accomplished:
+
+## Core Components Implemented
+
+### 1. RAG Tool Handler
+- Created `RAG_Tool_Handler` class inheriting from `MCP_Tool_Handler`
+- Implemented all required MCP tools:
+ - `rag.search_fts` - Keyword search using FTS5
+ - `rag.search_vector` - Semantic search using vector embeddings
+ - `rag.search_hybrid` - Hybrid search with two modes (fuse and fts_then_vec)
+ - `rag.get_chunks` - Fetch chunk content
+ - `rag.get_docs` - Fetch document content
+ - `rag.fetch_from_source` - Refetch authoritative data
+ - `rag.admin.stats` - Operational statistics
+
+### 2. Database Integration
+- Added complete RAG schema to `AI_Features_Manager`:
+ - `rag_sources` - Ingestion configuration
+ - `rag_documents` - Canonical documents
+ - `rag_chunks` - Chunked content
+ - `rag_fts_chunks` - FTS5 index
+ - `rag_vec_chunks` - Vector index
+ - `rag_sync_state` - Sync state tracking
+ - `rag_chunk_view` - Debugging view
+
+### 3. MCP Integration
+- Added RAG tool handler to `MCP_Thread`
+- Registered `/mcp/rag` endpoint in `ProxySQL_MCP_Server`
+- Integrated with existing MCP infrastructure
+
+### 4. Configuration
+- Added RAG configuration variables to `GenAI_Thread`:
+ - `genai_rag_enabled`
+ - `genai_rag_k_max`
+ - `genai_rag_candidates_max`
+ - `genai_rag_query_max_bytes`
+ - `genai_rag_response_max_bytes`
+ - `genai_rag_timeout_ms`
+
+## Key Features
+
+### Search Capabilities
+- **FTS Search**: Full-text search using SQLite FTS5
+- **Vector Search**: Semantic search using sqlite3-vec
+- **Hybrid Search**: Two modes:
+ - Fuse mode: Parallel FTS + vector with Reciprocal Rank Fusion
+ - FTS-then-vector mode: Candidate generation + rerank
+
+### Security Features
+- Input validation and sanitization
+- Query length limits
+- Result size limits
+- Timeouts for all operations
+- Column whitelisting for refetch operations
+- Row and byte limits
+
+### Performance Features
+- Proper use of prepared statements
+- Connection management
+- SQLite3-vec integration
+- FTS5 integration
+- Proper indexing strategies
+
+## Testing and Documentation
+
+### Test Scripts
+- `scripts/mcp/test_rag.sh` - Tests RAG functionality via MCP endpoint
+- `test/test_rag_schema.cpp` - Tests RAG database schema creation
+- `test/build_rag_test.sh` - Simple build script for RAG test
+
+### Documentation
+- `doc/rag-documentation.md` - Comprehensive RAG documentation
+- `doc/rag-examples.md` - Examples of using RAG tools
+- Updated `scripts/mcp/README.md` to include RAG in architecture
+
+## Files Created/Modified
+
+### New Files (10)
+1. `include/RAG_Tool_Handler.h` - Header file
+2. `lib/RAG_Tool_Handler.cpp` - Implementation file
+3. `doc/rag-documentation.md` - Documentation
+4. `doc/rag-examples.md` - Usage examples
+5. `scripts/mcp/test_rag.sh` - Test script
+6. `test/test_rag_schema.cpp` - Schema test
+7. `test/build_rag_test.sh` - Build script
+8. `RAG_IMPLEMENTATION_SUMMARY.md` - Implementation summary
+9. `RAG_FILE_SUMMARY.md` - File summary
+10. Updated `test/Makefile` - Added RAG test target
+
+### Modified Files (7)
+1. `include/MCP_Thread.h` - Added RAG tool handler member
+2. `lib/MCP_Thread.cpp` - Added initialization/cleanup
+3. `lib/ProxySQL_MCP_Server.cpp` - Registered RAG endpoint
+4. `lib/AI_Features_Manager.cpp` - Added RAG schema
+5. `include/GenAI_Thread.h` - Added RAG config variables
+6. `lib/GenAI_Thread.cpp` - Added RAG config initialization
+7. `scripts/mcp/README.md` - Updated documentation
+
+## Usage
+
+To enable RAG functionality:
+
+```sql
+-- Enable GenAI module
+SET genai.enabled = true;
+
+-- Enable RAG features
+SET genai.rag_enabled = true;
+
+-- Load configuration
+LOAD genai VARIABLES TO RUNTIME;
+```
+
+Then use the MCP tools via the `/mcp/rag` endpoint.
+
+## Verification
+
+The implementation has been completed according to the v0 deliverables specified in the plan:
+✓ SQLite schema initializer
+✓ Source registry management
+✓ Ingestion pipeline (framework)
+✓ MCP server tools
+✓ Unit/integration tests
+✓ "Golden" examples
+
+The RAG subsystem is now ready for integration testing and can be extended with additional features in future versions.
\ No newline at end of file
diff --git a/RAG_IMPLEMENTATION_SUMMARY.md b/RAG_IMPLEMENTATION_SUMMARY.md
new file mode 100644
index 0000000000..fea9a0c753
--- /dev/null
+++ b/RAG_IMPLEMENTATION_SUMMARY.md
@@ -0,0 +1,130 @@
+# ProxySQL RAG Subsystem Implementation - Complete
+
+## Implementation Status: COMPLETE
+
+I have successfully implemented the ProxySQL RAG (Retrieval-Augmented Generation) subsystem according to the requirements specified in the blueprint documents. Here's what has been accomplished:
+
+## Core Components Implemented
+
+### 1. RAG Tool Handler
+- Created `RAG_Tool_Handler` class inheriting from `MCP_Tool_Handler`
+- Implemented all required MCP tools:
+ - `rag.search_fts` - Keyword search using FTS5
+ - `rag.search_vector` - Semantic search using vector embeddings
+ - `rag.search_hybrid` - Hybrid search with two modes (fuse and fts_then_vec)
+ - `rag.get_chunks` - Fetch chunk content
+ - `rag.get_docs` - Fetch document content
+ - `rag.fetch_from_source` - Refetch authoritative data
+ - `rag.admin.stats` - Operational statistics
+
+### 2. Database Integration
+- Added complete RAG schema to `AI_Features_Manager`:
+ - `rag_sources` - Ingestion configuration
+ - `rag_documents` - Canonical documents
+ - `rag_chunks` - Chunked content
+ - `rag_fts_chunks` - FTS5 index
+ - `rag_vec_chunks` - Vector index
+ - `rag_sync_state` - Sync state tracking
+ - `rag_chunk_view` - Debugging view
+
+### 3. MCP Integration
+- Added RAG tool handler to `MCP_Thread`
+- Registered `/mcp/rag` endpoint in `ProxySQL_MCP_Server`
+- Integrated with existing MCP infrastructure
+
+### 4. Configuration
+- Added RAG configuration variables to `GenAI_Thread`:
+ - `genai_rag_enabled`
+ - `genai_rag_k_max`
+ - `genai_rag_candidates_max`
+ - `genai_rag_query_max_bytes`
+ - `genai_rag_response_max_bytes`
+ - `genai_rag_timeout_ms`
+
+## Key Features Implemented
+
+### Search Capabilities
+- **FTS Search**: Full-text search using SQLite FTS5
+- **Vector Search**: Semantic search using sqlite3-vec
+- **Hybrid Search**: Two modes:
+ - Fuse mode: Parallel FTS + vector with Reciprocal Rank Fusion
+ - FTS-then-vector mode: Candidate generation + rerank
+
+### Security Features
+- Input validation and sanitization
+- Query length limits
+- Result size limits
+- Timeouts for all operations
+- Column whitelisting for refetch operations
+- Row and byte limits
+
+### Performance Features
+- Proper use of prepared statements
+- Connection management
+- SQLite3-vec integration
+- FTS5 integration
+- Proper indexing strategies
+
+## Testing and Documentation
+
+### Test Scripts
+- `scripts/mcp/test_rag.sh` - Tests RAG functionality via MCP endpoint
+- `test/test_rag_schema.cpp` - Tests RAG database schema creation
+- `test/build_rag_test.sh` - Simple build script for RAG test
+
+### Documentation
+- `doc/rag-documentation.md` - Comprehensive RAG documentation
+- `doc/rag-examples.md` - Examples of using RAG tools
+- Updated `scripts/mcp/README.md` to include RAG in architecture
+
+## Files Created/Modified
+
+### New Files (10)
+1. `include/RAG_Tool_Handler.h` - Header file
+2. `lib/RAG_Tool_Handler.cpp` - Implementation file
+3. `doc/rag-documentation.md` - Documentation
+4. `doc/rag-examples.md` - Usage examples
+5. `scripts/mcp/test_rag.sh` - Test script
+6. `test/test_rag_schema.cpp` - Schema test
+7. `test/build_rag_test.sh` - Build script
+8. `RAG_IMPLEMENTATION_SUMMARY.md` - Implementation summary
+9. `RAG_FILE_SUMMARY.md` - File summary
+10. Updated `test/Makefile` - Added RAG test target
+
+### Modified Files (7)
+1. `include/MCP_Thread.h` - Added RAG tool handler member
+2. `lib/MCP_Thread.cpp` - Added initialization/cleanup
+3. `lib/ProxySQL_MCP_Server.cpp` - Registered RAG endpoint
+4. `lib/AI_Features_Manager.cpp` - Added RAG schema
+5. `include/GenAI_Thread.h` - Added RAG config variables
+6. `lib/GenAI_Thread.cpp` - Added RAG config initialization
+7. `scripts/mcp/README.md` - Updated documentation
+
+## Usage
+
+To enable RAG functionality:
+
+```sql
+-- Enable GenAI module
+SET genai.enabled = true;
+
+-- Enable RAG features
+SET genai.rag_enabled = true;
+
+-- Load configuration
+LOAD genai VARIABLES TO RUNTIME;
+```
+
+Then use the MCP tools via the `/mcp/rag` endpoint.
+
+## Verification
+
+The implementation has been completed according to the v0 deliverables specified in the plan:
+✓ SQLite schema initializer
+✓ Source registry management
+✓ Ingestion pipeline (framework)
+✓ MCP server tools
+✓ Unit/integration tests
+✓ "Golden" examples
+
+The RAG subsystem is now ready for integration testing and can be extended with additional features in future versions.
\ No newline at end of file
diff --git a/RAG_POC/architecture-data-model.md b/RAG_POC/architecture-data-model.md
new file mode 100644
index 0000000000..0c672bcee3
--- /dev/null
+++ b/RAG_POC/architecture-data-model.md
@@ -0,0 +1,384 @@
+# ProxySQL RAG Index — Data Model & Ingestion Architecture (v0 Blueprint)
+
+This document explains the SQLite data model used to turn relational tables (e.g. MySQL `posts`) into a retrieval-friendly index hosted inside ProxySQL. It focuses on:
+
+- What each SQLite table does
+- How tables relate to each other
+- How `rag_sources` defines **explicit mapping rules** (no guessing)
+- How ingestion transforms rows into documents and chunks
+- How FTS and vector indexes are maintained
+- What evolves later for incremental sync and updates
+
+---
+
+## 1. Goal and core idea
+
+Relational databases are excellent for structured queries, but RAG-style retrieval needs:
+
+- Fast keyword search (error messages, identifiers, tags)
+- Fast semantic search (similar meaning, paraphrased questions)
+- A stable way to “refetch the authoritative data” from the source DB
+
+The model below implements a **canonical document layer** inside ProxySQL:
+
+1. Ingest selected rows from a source database (MySQL, PostgreSQL, etc.)
+2. Convert each row into a **document** (title/body + metadata)
+3. Split long bodies into **chunks**
+4. Index chunks in:
+ - **FTS5** for keyword search
+ - **sqlite3-vec** for vector similarity
+5. Serve retrieval through stable APIs (MCP or SQL), independent of where indexes physically live in the future
+
+---
+
+## 2. The SQLite tables (what they are and why they exist)
+
+### 2.1 `rag_sources` — control plane: “what to ingest and how”
+
+**Purpose**
+- Defines each ingestion source (a table or view in an external DB)
+- Stores *explicit* transformation rules:
+ - which columns become `title`, `body`
+ - which columns go into `metadata_json`
+ - how to build `doc_id`
+- Stores chunking strategy and embedding strategy configuration
+
+**Key columns**
+- `backend_*`: how to connect (v0 connects directly; later may be “via ProxySQL”)
+- `table_name`, `pk_column`: what to ingest
+- `where_sql`: optional restriction (e.g. only questions)
+- `doc_map_json`: mapping rules (required)
+- `chunking_json`: chunking rules (required)
+- `embedding_json`: embedding rules (optional)
+
+**Important**: `rag_sources` is the **only place** that defines mapping logic.
+A general-purpose ingester must never “guess” which fields belong to `body` or metadata.
+
+---
+
+### 2.2 `rag_documents` — canonical documents: “one per source row”
+
+**Purpose**
+- Represents the canonical document created from a single source row.
+- Stores:
+ - a stable identifier (`doc_id`)
+ - a refetch pointer (`pk_json`)
+ - document text (`title`, `body`)
+ - structured metadata (`metadata_json`)
+
+**Why store full `body` here?**
+- Enables re-chunking later without re-fetching from the source DB.
+- Makes debugging and inspection easier.
+- Supports future update detection and diffing.
+
+**Key columns**
+- `doc_id` (PK): stable across runs and machines (e.g. `"posts:12345"`)
+- `source_id`: ties back to `rag_sources`
+- `pk_json`: how to refetch the authoritative row later (e.g. `{"Id":12345}`)
+- `title`, `body`: canonical text
+- `metadata_json`: non-text signals used for filters/boosting
+- `updated_at`, `deleted`: lifecycle fields for incremental sync later
+
+---
+
+### 2.3 `rag_chunks` — retrieval units: “one or many per document”
+
+**Purpose**
+- Stores chunked versions of a document’s text.
+- Retrieval and embeddings are performed at the chunk level for better quality.
+
+**Why chunk at all?**
+- Long bodies reduce retrieval quality:
+ - FTS returns large documents where only a small part is relevant
+ - Vector embeddings of large texts smear multiple topics together
+- Chunking yields:
+ - better precision
+ - better citations (“this chunk”) and smaller context
+ - cheaper updates (only re-embed changed chunks later)
+
+**Key columns**
+- `chunk_id` (PK): stable, derived from doc_id + chunk index (e.g. `"posts:12345#0"`)
+- `doc_id` (FK): parent document
+- `source_id`: convenience for filtering without joining documents
+- `chunk_index`: 0..N-1
+- `title`, `body`: chunk text (often title repeated for context)
+- `metadata_json`: optional chunk-level metadata (offsets, “has_code”, section label)
+- `updated_at`, `deleted`: lifecycle for later incremental sync
+
+---
+
+### 2.4 `rag_fts_chunks` — FTS5 index (contentless)
+
+**Purpose**
+- Keyword search index for chunks.
+- Best for:
+ - exact terms
+ - identifiers
+ - error messages
+ - tags and code tokens (depending on tokenization)
+
+**Design choice: contentless FTS**
+- The FTS virtual table does not automatically mirror `rag_chunks`.
+- The ingester explicitly inserts into FTS as chunks are created.
+- This makes ingestion deterministic and avoids surprises when chunk bodies change later.
+
+**Stored fields**
+- `chunk_id` (unindexed, acts like a row identifier)
+- `title`, `body` (indexed)
+
+---
+
+### 2.5 `rag_vec_chunks` — vector index (sqlite3-vec)
+
+**Purpose**
+- Semantic similarity search over chunks.
+- Each chunk has a vector embedding.
+
+**Key columns**
+- `embedding float[DIM]`: embedding vector (DIM must match your model)
+- `chunk_id`: join key to `rag_chunks`
+- Optional metadata columns:
+ - `doc_id`, `source_id`, `updated_at`
+ - These help filtering and joining and are valuable for performance.
+
+**Note**
+- The ingester decides what text is embedded (chunk body alone, or “Title + Tags + Body chunk”).
+
+---
+
+### 2.6 Optional convenience objects
+- `rag_chunk_view`: joins `rag_chunks` with `rag_documents` for debugging/inspection
+- `rag_sync_state`: reserved for incremental sync later (not used in v0)
+
+---
+
+## 3. Table relationships (the graph)
+
+Think of this as a data pipeline graph:
+
+```text
+rag_sources
+ (defines mapping + chunking + embedding)
+ |
+ v
+rag_documents (1 row per source row)
+ |
+ v
+rag_chunks (1..N chunks per document)
+ / \
+ v v
+rag_fts rag_vec
+```
+
+**Cardinality**
+- `rag_sources (1) -> rag_documents (N)`
+- `rag_documents (1) -> rag_chunks (N)`
+- `rag_chunks (1) -> rag_fts_chunks (1)` (insertion done by ingester)
+- `rag_chunks (1) -> rag_vec_chunks (0/1+)` (0 if embeddings disabled; 1 typically)
+
+---
+
+## 4. How mapping is defined (no guessing)
+
+### 4.1 Why `doc_map_json` exists
+A general-purpose system cannot infer that:
+- `posts.Body` should become document body
+- `posts.Title` should become title
+- `Score`, `Tags`, `CreationDate`, etc. should become metadata
+- Or how to concatenate fields
+
+Therefore, `doc_map_json` is required.
+
+### 4.2 `doc_map_json` structure (v0)
+`doc_map_json` defines:
+
+- `doc_id.format`: string template with `{ColumnName}` placeholders
+- `title.concat`: concatenation spec
+- `body.concat`: concatenation spec
+- `metadata.pick`: list of column names to include in metadata JSON
+- `metadata.rename`: mapping of old key -> new key (useful for typos or schema differences)
+
+**Concatenation parts**
+- `{"col":"Column"}` — appends the column value (if present)
+- `{"lit":"..."} ` — appends a literal string
+
+Example (posts-like):
+
+```json
+{
+ "doc_id": { "format": "posts:{Id}" },
+ "title": { "concat": [ { "col": "Title" } ] },
+ "body": { "concat": [ { "col": "Body" } ] },
+ "metadata": {
+ "pick": ["Id","PostTypeId","Tags","Score","CreaionDate"],
+ "rename": {"CreaionDate":"CreationDate"}
+ }
+}
+```
+
+---
+
+## 5. Chunking strategy definition
+
+### 5.1 Why chunking is configured per source
+Different tables need different chunking:
+- StackOverflow `Body` may be long -> chunking recommended
+- Small “reference” tables may not need chunking at all
+
+Thus chunking is stored in `rag_sources.chunking_json`.
+
+### 5.2 `chunking_json` structure (v0)
+v0 supports **chars-based** chunking (simple, robust).
+
+```json
+{
+ "enabled": true,
+ "unit": "chars",
+ "chunk_size": 4000,
+ "overlap": 400,
+ "min_chunk_size": 800
+}
+```
+
+**Behavior**
+- If `body.length <= chunk_size` -> one chunk
+- Else chunks of `chunk_size` with `overlap`
+- Avoid tiny final chunks by appending the tail to the previous chunk if below `min_chunk_size`
+
+**Why overlap matters**
+- Prevents splitting a key sentence or code snippet across boundaries
+- Improves both FTS and semantic retrieval consistency
+
+---
+
+## 6. Embedding strategy definition (where it fits in the model)
+
+### 6.1 Why embeddings are per chunk
+- Better retrieval precision
+- Smaller context per match
+- Allows partial updates later (only re-embed changed chunks)
+
+### 6.2 `embedding_json` structure (v0)
+```json
+{
+ "enabled": true,
+ "dim": 1536,
+ "model": "text-embedding-3-large",
+ "input": { "concat": [
+ {"col":"Title"},
+ {"lit":"\nTags: "}, {"col":"Tags"},
+ {"lit":"\n\n"},
+ {"chunk_body": true}
+ ]}
+}
+```
+
+**Meaning**
+- Build embedding input text from:
+ - title
+ - tags (as plain text)
+ - chunk body
+
+This improves semantic retrieval for question-like content without embedding numeric metadata.
+
+---
+
+## 7. Ingestion lifecycle (step-by-step)
+
+For each enabled `rag_sources` entry:
+
+1. **Connect** to source DB using `backend_*`
+2. **Select rows** from `table_name` (and optional `where_sql`)
+ - Select only needed columns determined by `doc_map_json` and `embedding_json`
+3. For each row:
+ - Build `doc_id` using `doc_map_json.doc_id.format`
+ - Build `pk_json` from `pk_column`
+ - Build `title` using `title.concat`
+ - Build `body` using `body.concat`
+ - Build `metadata_json` using `metadata.pick` and `metadata.rename`
+4. **Skip** if `doc_id` already exists (v0 behavior)
+5. Insert into `rag_documents`
+6. Chunk `body` using `chunking_json`
+7. For each chunk:
+ - Insert into `rag_chunks`
+ - Insert into `rag_fts_chunks`
+ - If embeddings enabled:
+ - Build embedding input text using `embedding_json.input`
+ - Compute embedding
+ - Insert into `rag_vec_chunks`
+8. Commit (ideally in a transaction for performance)
+
+---
+
+## 8. What changes later (incremental sync and updates)
+
+v0 is “insert-only and skip-existing.”
+Product-grade ingestion requires:
+
+### 8.1 Detecting changes
+Options:
+- Watermark by `LastActivityDate` / `updated_at` column
+- Hash (e.g. `sha256(title||body||metadata)`) stored in documents table
+- Compare chunk hashes to re-embed only changed chunks
+
+### 8.2 Updating and deleting
+Needs:
+- Upsert documents
+- Delete or mark `deleted=1` when source row deleted
+- Rebuild chunks and indexes when body changes
+- Maintain FTS rows:
+ - delete old chunk rows from FTS
+ - insert updated chunk rows
+
+### 8.3 Checkpoints
+Use `rag_sync_state` to store:
+- last ingested timestamp
+- GTID/LSN for CDC
+- or a monotonic PK watermark
+
+The current schema already includes:
+- `updated_at` and `deleted`
+- `rag_sync_state` placeholder
+
+So incremental sync can be added without breaking the data model.
+
+---
+
+## 9. Practical example: mapping `posts` table
+
+Given a MySQL `posts` row:
+
+- `Id = 12345`
+- `Title = "How to parse JSON in MySQL 8?"`
+- `Body = "I tried JSON_EXTRACT...
"`
+- `Tags = ""`
+- `Score = 12`
+
+With mapping:
+
+- `doc_id = "posts:12345"`
+- `title = Title`
+- `body = Body`
+- `metadata_json` includes `{ "Tags": "...", "Score": "12", ... }`
+- chunking splits body into:
+ - `posts:12345#0`, `posts:12345#1`, etc.
+- FTS is populated with the chunk text
+- vectors are stored per chunk
+
+---
+
+## 10. Summary
+
+This data model separates concerns cleanly:
+
+- `rag_sources` defines *policy* (what/how to ingest)
+- `rag_documents` defines canonical *identity and refetch pointer*
+- `rag_chunks` defines retrieval *units*
+- `rag_fts_chunks` defines keyword search
+- `rag_vec_chunks` defines semantic search
+
+This separation makes the system:
+- general purpose (works for many schemas)
+- deterministic (no magic inference)
+- extensible to incremental sync, external indexes, and richer hybrid retrieval
+
diff --git a/RAG_POC/architecture-runtime-retrieval.md b/RAG_POC/architecture-runtime-retrieval.md
new file mode 100644
index 0000000000..8f033e5301
--- /dev/null
+++ b/RAG_POC/architecture-runtime-retrieval.md
@@ -0,0 +1,344 @@
+# ProxySQL RAG Engine — Runtime Retrieval Architecture (v0 Blueprint)
+
+This document describes how ProxySQL becomes a **RAG retrieval engine** at runtime. The companion document (Data Model & Ingestion) explains how content enters the SQLite index. This document explains how content is **queried**, how results are **returned to agents/applications**, and how **hybrid retrieval** works in practice.
+
+It is written as an implementation blueprint for ProxySQL (and its MCP server) and assumes the SQLite schema contains:
+
+- `rag_sources` (control plane)
+- `rag_documents` (canonical docs)
+- `rag_chunks` (retrieval units)
+- `rag_fts_chunks` (FTS5)
+- `rag_vec_chunks` (sqlite3-vec vectors)
+
+---
+
+## 1. The runtime role of ProxySQL in a RAG system
+
+ProxySQL becomes a RAG runtime by providing four capabilities in one bounded service:
+
+1. **Retrieval Index Host**
+ - Hosts the SQLite index and search primitives (FTS + vectors).
+ - Offers deterministic query semantics and strict budgets.
+
+2. **Orchestration Layer**
+ - Implements search flows (FTS, vector, hybrid, rerank).
+ - Applies filters, caps, and result shaping.
+
+3. **Stable API Surface (MCP-first)**
+ - LLM agents call MCP tools (not raw SQL).
+ - Tool contracts remain stable even if internal storage changes.
+
+4. **Authoritative Row Refetch Gateway**
+ - After retrieval returns `doc_id` / `pk_json`, ProxySQL can refetch the authoritative row from the source DB on-demand (optional).
+ - This avoids returning stale or partial data when the full row is needed.
+
+In production terms, this is not “ProxySQL as a general search engine.” It is a **bounded retrieval service** colocated with database access logic.
+
+---
+
+## 2. High-level query flow (agent-centric)
+
+A typical RAG flow has two phases:
+
+### Phase A — Retrieval (fast, bounded, cheap)
+- Query the index to obtain a small number of relevant chunks (and their parent doc identity).
+- Output includes `chunk_id`, `doc_id`, `score`, and small metadata.
+
+### Phase B — Fetch (optional, authoritative, bounded)
+- If the agent needs full context or structured fields, it refetches the authoritative row from the source DB using `pk_json`.
+- This avoids scanning large tables and avoids shipping huge payloads in Phase A.
+
+**Canonical flow**
+1. `rag.search_hybrid(query, filters, k)` → returns top chunk ids and scores
+2. `rag.get_chunks(chunk_ids)` → returns chunk text for prompt grounding/citations
+3. Optional: `rag.fetch_from_source(doc_id)` → returns full row or selected columns
+
+---
+
+## 3. Runtime interfaces: MCP vs SQL
+
+ProxySQL should support two “consumption modes”:
+
+### 3.1 MCP tools (preferred for AI agents)
+- Strict limits and predictable response schemas.
+- Tools return structured results and avoid SQL injection concerns.
+- Agents do not need direct DB access.
+
+### 3.2 SQL access (for standard applications / debugging)
+- Applications may connect to ProxySQL’s SQLite admin interface (or a dedicated port) and issue SQL.
+- Useful for:
+ - internal dashboards
+ - troubleshooting
+ - non-agent apps that want retrieval but speak SQL
+
+**Principle**
+- MCP is the stable, long-term interface.
+- SQL is optional and may be restricted to trusted callers.
+
+---
+
+## 4. Retrieval primitives
+
+### 4.1 FTS retrieval (keyword / exact match)
+
+FTS5 is used for:
+- error messages
+- identifiers and function names
+- tags and exact terms
+- “grep-like” queries
+
+**Typical output**
+- `chunk_id`, `score_fts`, optional highlights/snippets
+
+**Ranking**
+- `bm25(rag_fts_chunks)` is the default. It is fast and effective for term queries.
+
+### 4.2 Vector retrieval (semantic similarity)
+
+Vector search is used for:
+- paraphrased questions
+- semantic similarity (“how to do X” vs “best way to achieve X”)
+- conceptual matching that is poor with keyword-only search
+
+**Typical output**
+- `chunk_id`, `score_vec` (distance/similarity), plus join metadata
+
+**Important**
+- Vectors are generally computed per chunk.
+- Filters are applied via `source_id` and joins to `rag_chunks` / `rag_documents`.
+
+---
+
+## 5. Hybrid retrieval patterns (two recommended modes)
+
+Hybrid retrieval combines FTS and vector search for better quality than either alone. Two concrete modes should be implemented because they solve different problems.
+
+### Mode 1 — “Best of both” (parallel FTS + vector; fuse results)
+**Use when**
+- the query may contain both exact tokens (e.g. error messages) and semantic intent
+
+**Flow**
+1. Run FTS top-N (e.g. N=50)
+2. Run vector top-N (e.g. N=50)
+3. Merge results by `chunk_id`
+4. Score fusion (recommended): Reciprocal Rank Fusion (RRF)
+5. Return top-k (e.g. k=10)
+
+**Why RRF**
+- Robust without score calibration
+- Works across heterogeneous score ranges (bm25 vs cosine distance)
+
+**RRF formula**
+- For each candidate chunk:
+ - `score = w_fts/(k0 + rank_fts) + w_vec/(k0 + rank_vec)`
+ - Typical: `k0=60`, `w_fts=1.0`, `w_vec=1.0`
+
+### Mode 2 — “Broad FTS then vector refine” (candidate generation + rerank)
+**Use when**
+- you want strong precision anchored to exact term matches
+- you want to avoid vector search over the entire corpus
+
+**Flow**
+1. Run broad FTS query top-M (e.g. M=200)
+2. Fetch chunk texts for those candidates
+3. Compute vector similarity of query embedding to candidate embeddings
+4. Return top-k
+
+This mode behaves like a two-stage retrieval pipeline:
+- Stage 1: cheap recall (FTS)
+- Stage 2: precise semantic rerank within candidates
+
+---
+
+## 6. Filters, constraints, and budgets (blast-radius control)
+
+A RAG retrieval engine must be bounded. ProxySQL should enforce limits at the MCP layer and ideally also at SQL helper functions.
+
+### 6.1 Hard caps (recommended defaults)
+- Maximum `k` returned: 50
+- Maximum candidates for broad-stage: 200–500
+- Maximum query length: e.g. 2–8 KB
+- Maximum response bytes: e.g. 1–5 MB
+- Maximum execution time per request: e.g. 50–250 ms for retrieval, 1–2 s for fetch
+
+### 6.2 Filter semantics
+Filters should be applied consistently across retrieval modes.
+
+Common filters:
+- `source_id` or `source_name`
+- tag include/exclude (via metadata_json parsing or pre-extracted tag fields later)
+- post type (question vs answer)
+- minimum score
+- time range (creation date / last activity)
+
+Implementation note:
+- v0 stores metadata in JSON; filtering can be implemented in MCP layer or via SQLite JSON functions (if enabled).
+- For performance, later versions should denormalize key metadata into dedicated columns or side tables.
+
+---
+
+## 7. Result shaping and what the caller receives
+
+A retrieval response must be designed for downstream LLM usage:
+
+### 7.1 Retrieval results (Phase A)
+Return a compact list of “evidence candidates”:
+
+- `chunk_id`
+- `doc_id`
+- `scores` (fts, vec, fused)
+- short `title`
+- minimal metadata (source, tags, timestamp, etc.)
+
+Do **not** return full bodies by default; that is what `rag.get_chunks` is for.
+
+### 7.2 Chunk fetch results (Phase A.2)
+`rag.get_chunks(chunk_ids)` returns:
+
+- `chunk_id`, `doc_id`
+- `title`
+- `body` (chunk text)
+- optionally a snippet/highlight for display
+
+### 7.3 Source refetch results (Phase B)
+`rag.fetch_from_source(doc_id)` returns:
+- either the full row
+- or a selected subset of columns (recommended)
+
+This is the “authoritative fetch” boundary that prevents stale/partial index usage from being a correctness problem.
+
+---
+
+## 8. SQL examples (runtime extraction)
+
+These are not the preferred agent interface, but they are crucial for debugging and for SQL-native apps.
+
+### 8.1 FTS search (top 10)
+```sql
+SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts
+FROM rag_fts_chunks f
+WHERE rag_fts_chunks MATCH 'json_extract mysql'
+ORDER BY score_fts
+LIMIT 10;
+```
+
+Join to fetch text:
+```sql
+SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts,
+ c.doc_id,
+ c.body
+FROM rag_fts_chunks f
+JOIN rag_chunks c ON c.chunk_id = f.chunk_id
+WHERE rag_fts_chunks MATCH 'json_extract mysql'
+ORDER BY score_fts
+LIMIT 10;
+```
+
+### 8.2 Vector search (top 10)
+Vector syntax depends on how you expose query vectors. A typical pattern is:
+
+1) Bind a query vector into a function / parameter
+2) Use `rag_vec_chunks` to return nearest neighbors
+
+Example shape (conceptual):
+```sql
+-- Pseudocode: nearest neighbors for :query_embedding
+SELECT
+ v.chunk_id,
+ v.distance
+FROM rag_vec_chunks v
+WHERE v.embedding MATCH :query_embedding
+ORDER BY v.distance
+LIMIT 10;
+```
+
+In production, ProxySQL MCP will typically compute the query embedding and call SQL internally with a bound parameter.
+
+---
+
+## 9. MCP tools (runtime API surface)
+
+This document does not define full schemas (that is in `mcp-tools.md`), but it defines what each tool must do.
+
+### 9.1 Retrieval
+- `rag.search_fts(query, filters, k)`
+- `rag.search_vector(query_text | query_embedding, filters, k)`
+- `rag.search_hybrid(query, mode, filters, k, params)`
+ - Mode 1: parallel + RRF fuse
+ - Mode 2: broad FTS candidates + vector rerank
+
+### 9.2 Fetch
+- `rag.get_chunks(chunk_ids)`
+- `rag.get_docs(doc_ids)`
+- `rag.fetch_from_source(doc_ids | pk_json, columns?, limits?)`
+
+**MCP-first principle**
+- Agents do not see SQLite schema or SQL.
+- MCP tools remain stable even if you move index storage out of ProxySQL later.
+
+---
+
+## 10. Operational considerations
+
+### 10.1 Dedicated ProxySQL instance
+Run GenAI retrieval in a dedicated ProxySQL instance to reduce blast radius:
+- independent CPU/memory budgets
+- independent configuration and rate limits
+- independent failure domain
+
+### 10.2 Observability and metrics (minimum)
+- count of docs/chunks per source
+- query counts by tool and source
+- p50/p95 latency for:
+ - FTS
+ - vector
+ - hybrid
+ - refetch
+- dropped/limited requests (rate limit hit, cap exceeded)
+- error rate and error categories
+
+### 10.3 Safety controls
+- strict upper bounds on `k` and candidate sizes
+- strict timeouts
+- response size caps
+- optional allowlists for sources accessible to agents
+- tenant boundaries via filters (strongly recommended for multi-tenant)
+
+---
+
+## 11. Recommended “v0-to-v1” evolution checklist
+
+### v0 (PoC)
+- ingestion to docs/chunks
+- FTS search
+- vector search (if embedding pipeline available)
+- simple hybrid search
+- chunk fetch
+- manual/limited source refetch
+
+### v1 (product hardening)
+- incremental sync checkpoints (`rag_sync_state`)
+- update detection (hashing/versioning)
+- delete handling
+- robust hybrid search:
+ - RRF fuse
+ - candidate-generation rerank
+- stronger filtering semantics (denormalized metadata columns)
+- quotas, rate limits, per-source budgets
+- full MCP tool contracts + tests
+
+---
+
+## 12. Summary
+
+At runtime, ProxySQL RAG retrieval is implemented as:
+
+- **Index query** (FTS/vector/hybrid) returning a small set of chunk IDs
+- **Chunk fetch** returning the text that the LLM will ground on
+- Optional **authoritative refetch** from the source DB by primary key
+- Strict limits and consistent filtering to keep the service bounded
+
diff --git a/RAG_POC/embeddings-design.md b/RAG_POC/embeddings-design.md
new file mode 100644
index 0000000000..796a06a570
--- /dev/null
+++ b/RAG_POC/embeddings-design.md
@@ -0,0 +1,353 @@
+# ProxySQL RAG Index — Embeddings & Vector Retrieval Design (Chunk-Level) (v0→v1 Blueprint)
+
+This document specifies how embeddings should be produced, stored, updated, and queried for chunk-level vector search in ProxySQL’s RAG index. It is intended as an implementation blueprint.
+
+It assumes:
+- Chunking is already implemented (`rag_chunks`).
+- ProxySQL includes **sqlite3-vec** and uses a `vec0(...)` virtual table (`rag_vec_chunks`).
+- Retrieval is exposed primarily via MCP tools (`mcp-tools.md`).
+
+---
+
+## 1. Design objectives
+
+1. **Chunk-level embeddings**
+ - Each chunk receives its own embedding for retrieval precision.
+
+2. **Deterministic embedding input**
+ - The text embedded is explicitly defined per source, not inferred.
+
+3. **Model agility**
+ - The system can change embedding models/dimensions without breaking stored data or APIs.
+
+4. **Efficient updates**
+ - Only recompute embeddings for chunks whose embedding input changed.
+
+5. **Operational safety**
+ - Bound cost and latency (embedding generation can be expensive).
+ - Allow asynchronous embedding jobs if needed later.
+
+---
+
+## 2. What to embed (and what not to embed)
+
+### 2.1 Embed text that improves semantic retrieval
+Recommended embedding input per chunk:
+
+- Document title (if present)
+- Tags (as plain text)
+- Chunk body
+
+Example embedding input template:
+```
+{Title}
+Tags: {Tags}
+
+{ChunkBody}
+```
+
+This typically improves semantic recall significantly for knowledge-base-like content (StackOverflow posts, docs, tickets, runbooks).
+
+### 2.2 Do NOT embed numeric metadata by default
+Do not embed fields like `Score`, `ViewCount`, `OwnerUserId`, timestamps, etc. These should remain structured and be used for:
+- filtering
+- boosting
+- tie-breaking
+- result shaping
+
+Embedding numeric metadata into text typically adds noise and reduces semantic quality.
+
+### 2.3 Code and HTML considerations
+If your chunk body contains HTML or code:
+- **v0**: embed raw text (works, but may be noisy)
+- **v1**: normalize to improve quality:
+ - strip HTML tags (keep text content)
+ - preserve code blocks as text, but consider stripping excessive markup
+ - optionally create specialized “code-only” chunks for code-heavy sources
+
+Normalization should be source-configurable.
+
+---
+
+## 3. Where embedding input rules are defined
+
+Embedding input rules must be explicit and stored per source.
+
+### 3.1 `rag_sources.embedding_json`
+Recommended schema:
+```json
+{
+ "enabled": true,
+ "model": "text-embedding-3-large",
+ "dim": 1536,
+ "input": {
+ "concat": [
+ {"col":"Title"},
+ {"lit":"\nTags: "}, {"col":"Tags"},
+ {"lit":"\n\n"},
+ {"chunk_body": true}
+ ]
+ },
+ "normalize": {
+ "strip_html": true,
+ "collapse_whitespace": true
+ }
+}
+```
+
+**Semantics**
+- `enabled`: whether to compute/store embeddings for this source
+- `model`: logical name (for observability and compatibility checks)
+- `dim`: vector dimension
+- `input.concat`: how to build embedding input text
+- `normalize`: optional normalization steps
+
+---
+
+## 4. Storage schema and model/versioning
+
+### 4.1 Current v0 schema: single vector table
+`rag_vec_chunks` stores:
+- embedding vector
+- chunk_id
+- doc_id/source_id convenience columns
+- updated_at
+
+This is appropriate for v0 when you assume a single embedding model/dimension.
+
+### 4.2 Recommended v1 evolution: support multiple models
+In a product setting, you may want multiple embedding models (e.g. general vs code-centric).
+
+Two ways to support this:
+
+#### Option A: include model identity columns in `rag_vec_chunks`
+Add columns:
+- `model TEXT`
+- `dim INTEGER` (optional if fixed per model)
+
+Then allow multiple rows per `chunk_id` (unique key becomes `(chunk_id, model)`).
+This may require schema change and a different vec0 design (some vec0 configurations support metadata columns, but uniqueness must be handled carefully).
+
+#### Option B: one vec table per model (recommended if vec0 constraints exist)
+Create:
+- `rag_vec_chunks_1536_v1`
+- `rag_vec_chunks_1024_code_v1`
+etc.
+
+Then MCP tools select the table based on requested model or default configuration.
+
+**Recommendation**
+Start with Option A only if your sqlite3-vec build makes it easy to filter by model. Otherwise, Option B is operationally cleaner.
+
+---
+
+## 5. Embedding generation pipeline
+
+### 5.1 When embeddings are created
+Embeddings are created during ingestion, immediately after chunk creation, if `embedding_json.enabled=true`.
+
+This provides a simple, synchronous pipeline:
+- ingest row → create chunks → compute embedding → store vector
+
+### 5.2 When embeddings should be updated
+Embeddings must be recomputed if the *embedding input string* changes. That depends on:
+- title changes
+- tags changes
+- chunk body changes
+- normalization rules changes (strip_html etc.)
+- embedding model changes
+
+Therefore, update logic should be based on a **content hash** of the embedding input.
+
+---
+
+## 6. Content hashing for efficient updates (v1 recommendation)
+
+### 6.1 Why hashing is needed
+Without hashing, you might recompute embeddings unnecessarily:
+- expensive
+- slow
+- prevents incremental sync from being efficient
+
+### 6.2 Recommended approach
+Store `embedding_input_hash` per chunk per model.
+
+Implementation options:
+
+#### Option A: Store hash in `rag_chunks.metadata_json`
+Example:
+```json
+{
+ "chunk_index": 0,
+ "embedding_hash": "sha256:...",
+ "embedding_model": "text-embedding-3-large"
+}
+```
+
+Pros: no schema changes.
+Cons: JSON parsing overhead.
+
+#### Option B: Dedicated side table (recommended)
+Create `rag_chunk_embedding_state`:
+
+```sql
+CREATE TABLE rag_chunk_embedding_state (
+ chunk_id TEXT NOT NULL,
+ model TEXT NOT NULL,
+ dim INTEGER NOT NULL,
+ input_hash TEXT NOT NULL,
+ updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+ PRIMARY KEY(chunk_id, model)
+);
+```
+
+Pros: fast lookups; avoids JSON parsing.
+Cons: extra table.
+
+**Recommendation**
+Use Option B for v1.
+
+---
+
+## 7. Embedding model integration options
+
+### 7.1 External embedding service (recommended initially)
+ProxySQL calls an embedding service:
+- OpenAI-compatible endpoint, or
+- local service (e.g. llama.cpp server), or
+- vendor-specific embedding API
+
+Pros:
+- easy to iterate on model choice
+- isolates ML runtime from ProxySQL process
+
+Cons:
+- network latency; requires caching and timeouts
+
+### 7.2 Embedded model runtime inside ProxySQL
+ProxySQL links to an embedding runtime (llama.cpp, etc.)
+
+Pros:
+- no network dependency
+- predictable latency if tuned
+
+Cons:
+- increases memory footprint
+- needs careful resource controls
+
+**Recommendation**
+Start with an external embedding provider and keep a modular interface that can be swapped later.
+
+---
+
+## 8. Query embedding generation
+
+Vector search needs a query embedding. Do this in the MCP layer:
+
+1. Take `query_text`
+2. Apply query normalization (optional but recommended)
+3. Compute query embedding using the same model used for chunks
+4. Execute vector search SQL with a bound embedding vector
+
+**Do not**
+- accept arbitrary embedding vectors from untrusted callers without validation
+- allow unbounded query lengths
+
+---
+
+## 9. Vector search semantics
+
+### 9.1 Distance vs similarity
+Depending on the embedding model and vec search primitive, vector search may return:
+- cosine distance (lower is better)
+- cosine similarity (higher is better)
+- L2 distance (lower is better)
+
+**Recommendation**
+Normalize to a “higher is better” score in MCP responses:
+- if distance: `score_vec = 1 / (1 + distance)` or similar monotonic transform
+
+Keep raw distance in debug fields if needed.
+
+### 9.2 Filtering
+Filtering should be supported by:
+- `source_id` restriction
+- optional metadata filters (doc-level or chunk-level)
+
+In v0, filter by `source_id` is easiest because `rag_vec_chunks` stores `source_id` as metadata.
+
+---
+
+## 10. Hybrid retrieval integration
+
+Embeddings are one leg of hybrid retrieval. Two recommended hybrid modes are described in `mcp-tools.md`:
+
+1. **Fuse**: top-N FTS and top-N vector, merged by chunk_id, fused by RRF
+2. **FTS then vector**: broad FTS candidates then vector rerank within candidates
+
+Embeddings support both:
+- Fuse mode needs global vector search top-N.
+- Candidate mode needs vector search restricted to candidate chunk IDs.
+
+Candidate mode is often cheaper and more precise when the query includes strong exact tokens.
+
+---
+
+## 11. Operational controls
+
+### 11.1 Resource limits
+Embedding generation must be bounded by:
+- max chunk size embedded
+- max chunks embedded per document
+- per-source embedding rate limit
+- timeouts when calling embedding provider
+
+### 11.2 Batch embedding
+To improve throughput, embed in batches:
+- collect N chunks
+- send embedding request for N inputs
+- store results
+
+### 11.3 Backpressure and async embedding
+For v1, consider decoupling embedding generation from ingestion:
+- ingestion stores chunks
+- embedding worker processes “pending” chunks and fills vectors
+
+This allows:
+- ingestion to remain fast
+- embedding to scale independently
+- retries on embedding failures
+
+In this design, store a state record:
+- pending / ok / error
+- last error message
+- retry count
+
+---
+
+## 12. Recommended implementation steps (coding agent checklist)
+
+### v0 (synchronous embedding)
+1. Implement `embedding_json` parsing in ingester
+2. Build embedding input string for each chunk
+3. Call embedding provider (or use a stub in development)
+4. Insert vector rows into `rag_vec_chunks`
+5. Implement `rag.search_vector` MCP tool using query embedding + vector SQL
+
+### v1 (efficient incremental embedding)
+1. Add `rag_chunk_embedding_state` table
+2. Store `input_hash` per chunk per model
+3. Only re-embed if hash changed
+4. Add async embedding worker option
+5. Add metrics for embedding throughput and failures
+
+---
+
+## 13. Summary
+
+- Compute embeddings per chunk, not per document.
+- Define embedding input explicitly in `rag_sources.embedding_json`.
+- Store vectors in `rag_vec_chunks` (vec0).
+- For production, add hash-based update detection and optional async embedding workers.
+- Normalize vector scores in MCP responses and keep raw distance for debugging.
+
diff --git a/RAG_POC/mcp-tools.md b/RAG_POC/mcp-tools.md
new file mode 100644
index 0000000000..be3fd39b53
--- /dev/null
+++ b/RAG_POC/mcp-tools.md
@@ -0,0 +1,465 @@
+# MCP Tooling for ProxySQL RAG Engine (v0 Blueprint)
+
+This document defines the MCP tool surface for querying ProxySQL’s embedded RAG index. It is intended as a stable interface for AI agents. Internally, these tools query the SQLite schema described in `schema.sql` and the retrieval logic described in `architecture-runtime-retrieval.md`.
+
+**Design goals**
+- Stable tool contracts (do not break agents when internals change)
+- Strict bounds (prevent unbounded scans / large outputs)
+- Deterministic schemas (agents can reliably parse outputs)
+- Separation of concerns:
+ - Retrieval returns identifiers and scores
+ - Fetch returns content
+ - Optional refetch returns authoritative source rows
+
+---
+
+## 1. Conventions
+
+### 1.1 Identifiers
+- `doc_id`: stable document identifier (e.g. `posts:12345`)
+- `chunk_id`: stable chunk identifier (e.g. `posts:12345#0`)
+- `source_id` / `source_name`: corresponds to `rag_sources`
+
+### 1.2 Scores
+- FTS score: `score_fts` (bm25; lower is better in SQLite’s bm25 by default)
+- Vector score: `score_vec` (distance or similarity, depending on implementation)
+- Hybrid score: `score` (normalized fused score; higher is better)
+
+**Recommendation**
+Normalize scores in MCP layer so:
+- higher is always better for agent ranking
+- raw internal ranking can still be returned as `score_fts_raw`, `distance_raw`, etc. if helpful
+
+### 1.3 Limits and budgets (recommended defaults)
+All tools should enforce caps, regardless of caller input:
+- `k_max = 50`
+- `candidates_max = 500`
+- `query_max_bytes = 8192`
+- `response_max_bytes = 5_000_000`
+- `timeout_ms` (per tool): 250–2000ms depending on tool type
+
+Tools must return a `truncated` boolean if limits reduce output.
+
+---
+
+## 2. Shared filter model
+
+Many tools accept the same filter structure. This is intentionally simple in v0.
+
+### 2.1 Filter object
+```json
+{
+ "source_ids": [1,2],
+ "source_names": ["stack_posts"],
+ "doc_ids": ["posts:12345"],
+ "min_score": 5,
+ "post_type_ids": [1],
+ "tags_any": ["mysql","json"],
+ "tags_all": ["mysql","json"],
+ "created_after": "2022-01-01T00:00:00Z",
+ "created_before": "2025-01-01T00:00:00Z"
+}
+```
+
+**Notes**
+- In v0, most filters map to `metadata_json` values. Implementation can:
+ - filter in SQLite if JSON functions are available, or
+ - filter in MCP layer after initial retrieval (acceptable for small k/candidates)
+- For production, denormalize hot filters into dedicated columns for speed.
+
+### 2.2 Filter behavior
+- If both `source_ids` and `source_names` are provided, treat as intersection.
+- If no source filter is provided, default to all enabled sources **but** enforce a strict global budget.
+
+---
+
+## 3. Tool: `rag.search_fts`
+
+Keyword search over `rag_fts_chunks`.
+
+### 3.1 Request schema
+```json
+{
+ "query": "json_extract mysql",
+ "k": 10,
+ "offset": 0,
+ "filters": { },
+ "return": {
+ "include_title": true,
+ "include_metadata": true,
+ "include_snippets": false
+ }
+}
+```
+
+### 3.2 Semantics
+- Executes FTS query (MATCH) over indexed content.
+- Returns top-k chunk matches with scores and identifiers.
+- Does not return full chunk bodies unless `include_snippets` is requested (still bounded).
+
+### 3.3 Response schema
+```json
+{
+ "results": [
+ {
+ "chunk_id": "posts:12345#0",
+ "doc_id": "posts:12345",
+ "source_id": 1,
+ "source_name": "stack_posts",
+ "score_fts": 0.73,
+ "title": "How to parse JSON in MySQL 8?",
+ "metadata": { "Tags": "", "Score": "12" }
+ }
+ ],
+ "truncated": false,
+ "stats": {
+ "k_requested": 10,
+ "k_returned": 10,
+ "ms": 12
+ }
+}
+```
+
+---
+
+## 4. Tool: `rag.search_vector`
+
+Semantic search over `rag_vec_chunks`.
+
+### 4.1 Request schema (text input)
+```json
+{
+ "query_text": "How do I extract JSON fields in MySQL?",
+ "k": 10,
+ "filters": { },
+ "embedding": {
+ "model": "text-embedding-3-large"
+ }
+}
+```
+
+### 4.2 Request schema (precomputed vector)
+```json
+{
+ "query_embedding": {
+ "dim": 1536,
+ "values_b64": "AAAA..." // float32 array packed and base64 encoded
+ },
+ "k": 10,
+ "filters": { }
+}
+```
+
+### 4.3 Semantics
+- If `query_text` is provided, ProxySQL computes embedding internally (preferred for agents).
+- If `query_embedding` is provided, ProxySQL uses it directly (useful for advanced clients).
+- Returns nearest chunks by distance/similarity.
+
+### 4.4 Response schema
+```json
+{
+ "results": [
+ {
+ "chunk_id": "posts:9876#1",
+ "doc_id": "posts:9876",
+ "source_id": 1,
+ "source_name": "stack_posts",
+ "score_vec": 0.82,
+ "title": "Query JSON columns efficiently",
+ "metadata": { "Tags": "", "Score": "8" }
+ }
+ ],
+ "truncated": false,
+ "stats": {
+ "k_requested": 10,
+ "k_returned": 10,
+ "ms": 18
+ }
+}
+```
+
+---
+
+## 5. Tool: `rag.search_hybrid`
+
+Hybrid search combining FTS and vectors. Supports two modes:
+
+- **Mode A**: parallel FTS + vector, fuse results (RRF recommended)
+- **Mode B**: broad FTS candidate generation, then vector rerank
+
+### 5.1 Request schema (Mode A: fuse)
+```json
+{
+ "query": "json_extract mysql",
+ "k": 10,
+ "filters": { },
+ "mode": "fuse",
+ "fuse": {
+ "fts_k": 50,
+ "vec_k": 50,
+ "rrf_k0": 60,
+ "w_fts": 1.0,
+ "w_vec": 1.0
+ }
+}
+```
+
+### 5.2 Request schema (Mode B: candidates + rerank)
+```json
+{
+ "query": "json_extract mysql",
+ "k": 10,
+ "filters": { },
+ "mode": "fts_then_vec",
+ "fts_then_vec": {
+ "candidates_k": 200,
+ "rerank_k": 50,
+ "vec_metric": "cosine"
+ }
+}
+```
+
+### 5.3 Semantics (Mode A)
+1. Run FTS top `fts_k`
+2. Run vector top `vec_k`
+3. Merge candidates by `chunk_id`
+4. Compute fused score (RRF recommended)
+5. Return top `k`
+
+### 5.4 Semantics (Mode B)
+1. Run FTS top `candidates_k`
+2. Compute vector similarity within those candidates
+ - either by joining candidate chunk_ids to stored vectors, or
+ - by embedding candidate chunk text on the fly (not recommended)
+3. Return top `k` reranked results
+4. Optionally return debug info about candidate stages
+
+### 5.5 Response schema
+```json
+{
+ "results": [
+ {
+ "chunk_id": "posts:12345#0",
+ "doc_id": "posts:12345",
+ "source_id": 1,
+ "source_name": "stack_posts",
+ "score": 0.91,
+ "score_fts": 0.74,
+ "score_vec": 0.86,
+ "title": "How to parse JSON in MySQL 8?",
+ "metadata": { "Tags": "", "Score": "12" },
+ "debug": {
+ "rank_fts": 3,
+ "rank_vec": 6
+ }
+ }
+ ],
+ "truncated": false,
+ "stats": {
+ "mode": "fuse",
+ "k_requested": 10,
+ "k_returned": 10,
+ "ms": 27
+ }
+}
+```
+
+---
+
+## 6. Tool: `rag.get_chunks`
+
+Fetch chunk bodies by chunk_id. This is how agents obtain grounding text.
+
+### 6.1 Request schema
+```json
+{
+ "chunk_ids": ["posts:12345#0", "posts:9876#1"],
+ "return": {
+ "include_title": true,
+ "include_doc_metadata": true,
+ "include_chunk_metadata": true
+ }
+}
+```
+
+### 6.2 Response schema
+```json
+{
+ "chunks": [
+ {
+ "chunk_id": "posts:12345#0",
+ "doc_id": "posts:12345",
+ "title": "How to parse JSON in MySQL 8?",
+ "body": "I tried JSON_EXTRACT...
",
+ "doc_metadata": { "Tags": "", "Score": "12" },
+ "chunk_metadata": { "chunk_index": 0 }
+ }
+ ],
+ "truncated": false,
+ "stats": { "ms": 6 }
+}
+```
+
+**Hard limit recommendation**
+- Cap total returned chunk bytes to a safe maximum (e.g. 1–2 MB).
+
+---
+
+## 7. Tool: `rag.get_docs`
+
+Fetch full canonical documents by doc_id (not chunks). Useful for inspection or compact docs.
+
+### 7.1 Request schema
+```json
+{
+ "doc_ids": ["posts:12345"],
+ "return": {
+ "include_body": true,
+ "include_metadata": true
+ }
+}
+```
+
+### 7.2 Response schema
+```json
+{
+ "docs": [
+ {
+ "doc_id": "posts:12345",
+ "source_id": 1,
+ "source_name": "stack_posts",
+ "pk_json": { "Id": 12345 },
+ "title": "How to parse JSON in MySQL 8?",
+ "body": "...
",
+ "metadata": { "Tags": "", "Score": "12" }
+ }
+ ],
+ "truncated": false,
+ "stats": { "ms": 7 }
+}
+```
+
+---
+
+## 8. Tool: `rag.fetch_from_source`
+
+Refetch authoritative rows from the source DB using `doc_id` (via pk_json).
+
+### 8.1 Request schema
+```json
+{
+ "doc_ids": ["posts:12345"],
+ "columns": ["Id","Title","Body","Tags","Score"],
+ "limits": {
+ "max_rows": 10,
+ "max_bytes": 200000
+ }
+}
+```
+
+### 8.2 Semantics
+- Look up doc(s) in `rag_documents` to get `source_id` and `pk_json`
+- Resolve source connection from `rag_sources`
+- Execute a parameterized query by primary key
+- Return requested columns only
+- Enforce strict limits
+
+### 8.3 Response schema
+```json
+{
+ "rows": [
+ {
+ "doc_id": "posts:12345",
+ "source_name": "stack_posts",
+ "row": {
+ "Id": 12345,
+ "Title": "How to parse JSON in MySQL 8?",
+ "Score": 12
+ }
+ }
+ ],
+ "truncated": false,
+ "stats": { "ms": 22 }
+}
+```
+
+**Security note**
+- This tool must not allow arbitrary SQL.
+- Only allow fetching by primary key and a whitelist of columns.
+
+---
+
+## 9. Tool: `rag.admin.stats` (recommended)
+
+Operational visibility for dashboards and debugging.
+
+### 9.1 Request
+```json
+{}
+```
+
+### 9.2 Response
+```json
+{
+ "sources": [
+ {
+ "source_id": 1,
+ "source_name": "stack_posts",
+ "docs": 123456,
+ "chunks": 456789,
+ "last_sync": null
+ }
+ ],
+ "stats": { "ms": 5 }
+}
+```
+
+---
+
+## 10. Tool: `rag.admin.sync` (optional in v0; required in v1)
+
+Kicks ingestion for a source or all sources. In v0, ingestion may run as a separate process; in ProxySQL product form, this would trigger an internal job.
+
+### 10.1 Request
+```json
+{
+ "source_names": ["stack_posts"]
+}
+```
+
+### 10.2 Response
+```json
+{
+ "accepted": true,
+ "job_id": "sync-2026-01-19T10:00:00Z"
+}
+```
+
+---
+
+## 11. Implementation notes (what the coding agent should implement)
+
+1. **Input validation and caps** for every tool.
+2. **Consistent filtering** across FTS/vector/hybrid.
+3. **Stable scoring semantics** (higher-is-better recommended).
+4. **Efficient joins**:
+ - vector search returns chunk_ids; join to `rag_chunks`/`rag_documents` for metadata.
+5. **Hybrid modes**:
+ - Mode A (fuse): implement RRF
+ - Mode B (fts_then_vec): candidate set then vector rerank
+6. **Error model**:
+ - return structured errors with codes (e.g. `INVALID_ARGUMENT`, `LIMIT_EXCEEDED`, `INTERNAL`)
+7. **Observability**:
+ - return `stats.ms` in responses
+ - track tool usage counters and latency histograms
+
+---
+
+## 12. Summary
+
+These MCP tools define a stable retrieval interface:
+
+- Search: `rag.search_fts`, `rag.search_vector`, `rag.search_hybrid`
+- Fetch: `rag.get_chunks`, `rag.get_docs`, `rag.fetch_from_source`
+- Admin: `rag.admin.stats`, optionally `rag.admin.sync`
+
diff --git a/RAG_POC/rag_ingest.cpp b/RAG_POC/rag_ingest.cpp
new file mode 100644
index 0000000000..415ded4229
--- /dev/null
+++ b/RAG_POC/rag_ingest.cpp
@@ -0,0 +1,1009 @@
+// rag_ingest.cpp
+//
+// ------------------------------------------------------------
+// ProxySQL RAG Ingestion PoC (General-Purpose)
+// ------------------------------------------------------------
+//
+// What this program does (v0):
+// 1) Opens the SQLite "RAG index" database (schema.sql must already be applied).
+// 2) Reads enabled sources from rag_sources.
+// 3) For each source:
+// - Connects to MySQL (for now).
+// - Builds a SELECT that fetches only needed columns.
+// - For each row:
+// * Builds doc_id / title / body / metadata_json using doc_map_json.
+// * Chunks body using chunking_json.
+// * Inserts into:
+// rag_documents
+// rag_chunks
+// rag_fts_chunks (FTS5 contentless table)
+// * Optionally builds embedding input text using embedding_json and inserts
+// embeddings into rag_vec_chunks (sqlite3-vec) via a stub embedding provider.
+// - Skips docs that already exist (v0 requirement).
+//
+// Later (v1+):
+// - Add rag_sync_state usage for incremental ingestion (watermark/CDC).
+// - Add hashing to detect changed docs/chunks and update/reindex accordingly.
+// - Replace the embedding stub with a real embedding generator.
+//
+// ------------------------------------------------------------
+// Dependencies
+// ------------------------------------------------------------
+// - sqlite3
+// - MySQL client library (mysqlclient / libmysqlclient)
+// - nlohmann/json (single header json.hpp)
+//
+// Build example (Linux/macOS):
+// g++ -std=c++17 -O2 rag_ingest.cpp -o rag_ingest \
+// -lsqlite3 -lmysqlclient
+//
+// Usage:
+// ./rag_ingest /path/to/rag_index.sqlite
+//
+// Notes:
+// - This is a blueprint-grade PoC, written to be readable and modifiable.
+// - It uses a conservative JSON mapping language so ingestion is deterministic.
+// - It avoids advanced C++ patterns on purpose.
+//
+// ------------------------------------------------------------
+// Supported JSON Specs
+// ------------------------------------------------------------
+//
+// doc_map_json (required):
+// {
+// "doc_id": { "format": "posts:{Id}" },
+// "title": { "concat": [ {"col":"Title"} ] },
+// "body": { "concat": [ {"col":"Body"} ] },
+// "metadata": {
+// "pick": ["Id","Tags","Score","CreaionDate"],
+// "rename": {"CreaionDate":"CreationDate"}
+// }
+// }
+//
+// chunking_json (required, v0 chunks doc "body" only):
+// {
+// "enabled": true,
+// "unit": "chars", // v0 supports "chars" only
+// "chunk_size": 4000,
+// "overlap": 400,
+// "min_chunk_size": 800
+// }
+//
+// embedding_json (optional):
+// {
+// "enabled": true,
+// "dim": 1536,
+// "model": "text-embedding-3-large", // informational
+// "input": { "concat": [
+// {"col":"Title"},
+// {"lit":"\nTags: "}, {"col":"Tags"},
+// {"lit":"\n\n"},
+// {"chunk_body": true}
+// ]}
+// }
+//
+// ------------------------------------------------------------
+// sqlite3-vec binding note
+// ------------------------------------------------------------
+// sqlite3-vec "vec0(embedding float[N])" generally expects a vector value.
+// The exact binding format can vary by build/config of sqlite3-vec.
+// This program includes a "best effort" binder that binds a float array as a BLOB.
+// If your sqlite3-vec build expects a different representation (e.g. a function to
+// pack vectors), adapt bind_vec_embedding() accordingly.
+// ------------------------------------------------------------
+
+#include
+#include
+
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+#include
+
+#include "json.hpp"
+using json = nlohmann::json;
+
+// -------------------------
+// Small helpers
+// -------------------------
+
+static void fatal(const std::string& msg) {
+ std::cerr << "FATAL: " << msg << "\n";
+ std::exit(1);
+}
+
+static std::string str_or_empty(const char* p) {
+ return p ? std::string(p) : std::string();
+}
+
+static int sqlite_exec(sqlite3* db, const std::string& sql) {
+ char* err = nullptr;
+ int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &err);
+ if (rc != SQLITE_OK) {
+ std::string e = err ? err : "(unknown sqlite error)";
+ sqlite3_free(err);
+ std::cerr << "SQLite error: " << e << "\nSQL: " << sql << "\n";
+ }
+ return rc;
+}
+
+static std::string json_dump_compact(const json& j) {
+ // Compact output (no pretty printing) to keep storage small.
+ return j.dump();
+}
+
+// -------------------------
+// Data model
+// -------------------------
+
+struct RagSource {
+ int source_id = 0;
+ std::string name;
+ int enabled = 0;
+
+ // backend connection
+ std::string backend_type; // "mysql" for now
+ std::string host;
+ int port = 3306;
+ std::string user;
+ std::string pass;
+ std::string db;
+
+ // table
+ std::string table_name;
+ std::string pk_column;
+ std::string where_sql; // optional
+
+ // transformation config
+ json doc_map_json;
+ json chunking_json;
+ json embedding_json; // optional; may be null/object
+};
+
+struct ChunkingConfig {
+ bool enabled = true;
+ std::string unit = "chars"; // v0 only supports chars
+ int chunk_size = 4000;
+ int overlap = 400;
+ int min_chunk_size = 800;
+};
+
+struct EmbeddingConfig {
+ bool enabled = false;
+ int dim = 1536;
+ std::string model = "unknown";
+ json input_spec; // expects {"concat":[...]}
+};
+
+// A row fetched from MySQL, as a name->string map.
+typedef std::unordered_map RowMap;
+
+// -------------------------
+// JSON parsing
+// -------------------------
+
+static ChunkingConfig parse_chunking_json(const json& j) {
+ ChunkingConfig cfg;
+ if (!j.is_object()) return cfg;
+
+ if (j.contains("enabled")) cfg.enabled = j["enabled"].get();
+ if (j.contains("unit")) cfg.unit = j["unit"].get();
+ if (j.contains("chunk_size")) cfg.chunk_size = j["chunk_size"].get();
+ if (j.contains("overlap")) cfg.overlap = j["overlap"].get();
+ if (j.contains("min_chunk_size")) cfg.min_chunk_size = j["min_chunk_size"].get();
+
+ if (cfg.chunk_size <= 0) cfg.chunk_size = 4000;
+ if (cfg.overlap < 0) cfg.overlap = 0;
+ if (cfg.overlap >= cfg.chunk_size) cfg.overlap = cfg.chunk_size / 4;
+ if (cfg.min_chunk_size < 0) cfg.min_chunk_size = 0;
+
+ // v0 only supports chars
+ if (cfg.unit != "chars") {
+ std::cerr << "WARN: chunking_json.unit=" << cfg.unit
+ << " not supported in v0. Falling back to chars.\n";
+ cfg.unit = "chars";
+ }
+
+ return cfg;
+}
+
+static EmbeddingConfig parse_embedding_json(const json& j) {
+ EmbeddingConfig cfg;
+ if (!j.is_object()) return cfg;
+
+ if (j.contains("enabled")) cfg.enabled = j["enabled"].get();
+ if (j.contains("dim")) cfg.dim = j["dim"].get();
+ if (j.contains("model")) cfg.model = j["model"].get();
+ if (j.contains("input")) cfg.input_spec = j["input"];
+
+ if (cfg.dim <= 0) cfg.dim = 1536;
+ return cfg;
+}
+
+// -------------------------
+// Row access
+// -------------------------
+
+static std::optional row_get(const RowMap& row, const std::string& key) {
+ auto it = row.find(key);
+ if (it == row.end()) return std::nullopt;
+ return it->second;
+}
+
+// -------------------------
+// doc_id.format implementation
+// -------------------------
+// Replaces occurrences of {ColumnName} with the value from the row map.
+// Example: "posts:{Id}" -> "posts:12345"
+static std::string apply_format(const std::string& fmt, const RowMap& row) {
+ std::string out;
+ out.reserve(fmt.size() + 32);
+
+ for (size_t i = 0; i < fmt.size(); i++) {
+ char c = fmt[i];
+ if (c == '{') {
+ size_t j = fmt.find('}', i + 1);
+ if (j == std::string::npos) {
+ // unmatched '{' -> treat as literal
+ out.push_back(c);
+ continue;
+ }
+ std::string col = fmt.substr(i + 1, j - (i + 1));
+ auto v = row_get(row, col);
+ if (v.has_value()) out += v.value();
+ i = j; // jump past '}'
+ } else {
+ out.push_back(c);
+ }
+ }
+ return out;
+}
+
+// -------------------------
+// concat spec implementation
+// -------------------------
+// Supported elements in concat array:
+// {"col":"Title"} -> append row["Title"] if present
+// {"lit":"\n\n"} -> append literal
+// {"chunk_body": true} -> append chunk body (only in embedding_json input)
+//
+static std::string eval_concat(const json& concat_spec,
+ const RowMap& row,
+ const std::string& chunk_body,
+ bool allow_chunk_body) {
+ if (!concat_spec.is_array()) return "";
+
+ std::string out;
+ for (const auto& part : concat_spec) {
+ if (!part.is_object()) continue;
+
+ if (part.contains("col")) {
+ std::string col = part["col"].get();
+ auto v = row_get(row, col);
+ if (v.has_value()) out += v.value();
+ } else if (part.contains("lit")) {
+ out += part["lit"].get();
+ } else if (allow_chunk_body && part.contains("chunk_body")) {
+ bool yes = part["chunk_body"].get();
+ if (yes) out += chunk_body;
+ }
+ }
+ return out;
+}
+
+// -------------------------
+// metadata builder
+// -------------------------
+// metadata spec:
+// "metadata": { "pick":[...], "rename":{...} }
+static json build_metadata(const json& meta_spec, const RowMap& row) {
+ json meta = json::object();
+
+ if (meta_spec.is_object()) {
+ // pick fields
+ if (meta_spec.contains("pick") && meta_spec["pick"].is_array()) {
+ for (const auto& colv : meta_spec["pick"]) {
+ if (!colv.is_string()) continue;
+ std::string col = colv.get();
+ auto v = row_get(row, col);
+ if (v.has_value()) meta[col] = v.value();
+ }
+ }
+
+ // rename keys
+ if (meta_spec.contains("rename") && meta_spec["rename"].is_object()) {
+ std::vector> renames;
+ for (auto it = meta_spec["rename"].begin(); it != meta_spec["rename"].end(); ++it) {
+ if (!it.value().is_string()) continue;
+ renames.push_back({it.key(), it.value().get()});
+ }
+ for (size_t i = 0; i < renames.size(); i++) {
+ const std::string& oldk = renames[i].first;
+ const std::string& newk = renames[i].second;
+ if (meta.contains(oldk)) {
+ meta[newk] = meta[oldk];
+ meta.erase(oldk);
+ }
+ }
+ }
+ }
+
+ return meta;
+}
+
+// -------------------------
+// Chunking (chars-based)
+// -------------------------
+
+static std::vector chunk_text_chars(const std::string& text, const ChunkingConfig& cfg) {
+ std::vector chunks;
+
+ if (!cfg.enabled) {
+ chunks.push_back(text);
+ return chunks;
+ }
+
+ if ((int)text.size() <= cfg.chunk_size) {
+ chunks.push_back(text);
+ return chunks;
+ }
+
+ int step = cfg.chunk_size - cfg.overlap;
+ if (step <= 0) step = cfg.chunk_size;
+
+ for (int start = 0; start < (int)text.size(); start += step) {
+ int end = start + cfg.chunk_size;
+ if (end > (int)text.size()) end = (int)text.size();
+ int len = end - start;
+ if (len <= 0) break;
+
+ // Avoid tiny final chunk by appending it to the previous chunk
+ if (len < cfg.min_chunk_size && !chunks.empty()) {
+ chunks.back() += text.substr(start, len);
+ break;
+ }
+
+ chunks.push_back(text.substr(start, len));
+
+ if (end == (int)text.size()) break;
+ }
+
+ return chunks;
+}
+
+// -------------------------
+// MySQL helpers
+// -------------------------
+
+static MYSQL* mysql_connect_or_die(const RagSource& s) {
+ MYSQL* conn = mysql_init(nullptr);
+ if (!conn) fatal("mysql_init failed");
+
+ // Set utf8mb4 for safety with StackOverflow-like content
+ mysql_options(conn, MYSQL_SET_CHARSET_NAME, "utf8mb4");
+
+ if (!mysql_real_connect(conn,
+ s.host.c_str(),
+ s.user.c_str(),
+ s.pass.c_str(),
+ s.db.c_str(),
+ s.port,
+ nullptr,
+ 0)) {
+ std::string err = mysql_error(conn);
+ mysql_close(conn);
+ fatal("MySQL connect failed: " + err);
+ }
+ return conn;
+}
+
+static RowMap mysql_row_to_map(MYSQL_RES* res, MYSQL_ROW row) {
+ RowMap m;
+ unsigned int n = mysql_num_fields(res);
+ MYSQL_FIELD* fields = mysql_fetch_fields(res);
+
+ for (unsigned int i = 0; i < n; i++) {
+ const char* name = fields[i].name;
+ const char* val = row[i];
+ if (name) {
+ m[name] = str_or_empty(val);
+ }
+ }
+ return m;
+}
+
+// Collect columns used by doc_map_json + embedding_json so SELECT is minimal.
+// v0: we intentionally keep this conservative (include pk + all referenced col parts + metadata.pick).
+static void add_unique(std::vector& cols, const std::string& c) {
+ for (size_t i = 0; i < cols.size(); i++) {
+ if (cols[i] == c) return;
+ }
+ cols.push_back(c);
+}
+
+static void collect_cols_from_concat(std::vector& cols, const json& concat_spec) {
+ if (!concat_spec.is_array()) return;
+ for (const auto& part : concat_spec) {
+ if (part.is_object() && part.contains("col") && part["col"].is_string()) {
+ add_unique(cols, part["col"].get());
+ }
+ }
+}
+
+static std::vector collect_needed_columns(const RagSource& s, const EmbeddingConfig& ecfg) {
+ std::vector cols;
+ add_unique(cols, s.pk_column);
+
+ // title/body concat
+ if (s.doc_map_json.contains("title") && s.doc_map_json["title"].contains("concat"))
+ collect_cols_from_concat(cols, s.doc_map_json["title"]["concat"]);
+ if (s.doc_map_json.contains("body") && s.doc_map_json["body"].contains("concat"))
+ collect_cols_from_concat(cols, s.doc_map_json["body"]["concat"]);
+
+ // metadata.pick
+ if (s.doc_map_json.contains("metadata") && s.doc_map_json["metadata"].contains("pick")) {
+ const auto& pick = s.doc_map_json["metadata"]["pick"];
+ if (pick.is_array()) {
+ for (const auto& c : pick) if (c.is_string()) add_unique(cols, c.get());
+ }
+ }
+
+ // embedding input concat (optional)
+ if (ecfg.enabled && ecfg.input_spec.is_object() && ecfg.input_spec.contains("concat")) {
+ collect_cols_from_concat(cols, ecfg.input_spec["concat"]);
+ }
+
+ // doc_id.format: we do not try to parse all placeholders; best practice is doc_id uses pk only.
+ // If you want doc_id.format to reference other columns, include them in metadata.pick or concat.
+
+ return cols;
+}
+
+static std::string build_select_sql(const RagSource& s, const std::vector& cols) {
+ std::string sql = "SELECT ";
+ for (size_t i = 0; i < cols.size(); i++) {
+ if (i) sql += ", ";
+ sql += "`" + cols[i] + "`";
+ }
+ sql += " FROM `" + s.table_name + "`";
+ if (!s.where_sql.empty()) {
+ sql += " WHERE " + s.where_sql;
+ }
+ return sql;
+}
+
+// -------------------------
+// SQLite prepared statements (batched insertion)
+// -------------------------
+
+struct SqliteStmts {
+ sqlite3_stmt* doc_exists = nullptr;
+ sqlite3_stmt* ins_doc = nullptr;
+ sqlite3_stmt* ins_chunk = nullptr;
+ sqlite3_stmt* ins_fts = nullptr;
+ sqlite3_stmt* ins_vec = nullptr; // optional (only used if embedding enabled)
+};
+
+static void sqlite_prepare_or_die(sqlite3* db, sqlite3_stmt** st, const char* sql) {
+ if (sqlite3_prepare_v2(db, sql, -1, st, nullptr) != SQLITE_OK) {
+ fatal(std::string("SQLite prepare failed: ") + sqlite3_errmsg(db) + "\nSQL: " + sql);
+ }
+}
+
+static void sqlite_finalize_all(SqliteStmts& s) {
+ if (s.doc_exists) sqlite3_finalize(s.doc_exists);
+ if (s.ins_doc) sqlite3_finalize(s.ins_doc);
+ if (s.ins_chunk) sqlite3_finalize(s.ins_chunk);
+ if (s.ins_fts) sqlite3_finalize(s.ins_fts);
+ if (s.ins_vec) sqlite3_finalize(s.ins_vec);
+ s = SqliteStmts{};
+}
+
+static void sqlite_bind_text(sqlite3_stmt* st, int idx, const std::string& v) {
+ sqlite3_bind_text(st, idx, v.c_str(), -1, SQLITE_TRANSIENT);
+}
+
+// Best-effort binder for sqlite3-vec embeddings (float32 array).
+// If your sqlite3-vec build expects a different encoding, change this function only.
+static void bind_vec_embedding(sqlite3_stmt* st, int idx, const std::vector& emb) {
+ const void* data = (const void*)emb.data();
+ int bytes = (int)(emb.size() * sizeof(float));
+ sqlite3_bind_blob(st, idx, data, bytes, SQLITE_TRANSIENT);
+}
+
+// Check if doc exists
+static bool sqlite_doc_exists(SqliteStmts& ss, const std::string& doc_id) {
+ sqlite3_reset(ss.doc_exists);
+ sqlite3_clear_bindings(ss.doc_exists);
+
+ sqlite_bind_text(ss.doc_exists, 1, doc_id);
+
+ int rc = sqlite3_step(ss.doc_exists);
+ return (rc == SQLITE_ROW);
+}
+
+// Insert doc
+static void sqlite_insert_doc(SqliteStmts& ss,
+ int source_id,
+ const std::string& source_name,
+ const std::string& doc_id,
+ const std::string& pk_json,
+ const std::string& title,
+ const std::string& body,
+ const std::string& meta_json) {
+ sqlite3_reset(ss.ins_doc);
+ sqlite3_clear_bindings(ss.ins_doc);
+
+ sqlite_bind_text(ss.ins_doc, 1, doc_id);
+ sqlite3_bind_int(ss.ins_doc, 2, source_id);
+ sqlite_bind_text(ss.ins_doc, 3, source_name);
+ sqlite_bind_text(ss.ins_doc, 4, pk_json);
+ sqlite_bind_text(ss.ins_doc, 5, title);
+ sqlite_bind_text(ss.ins_doc, 6, body);
+ sqlite_bind_text(ss.ins_doc, 7, meta_json);
+
+ int rc = sqlite3_step(ss.ins_doc);
+ if (rc != SQLITE_DONE) {
+ fatal(std::string("SQLite insert rag_documents failed: ") + sqlite3_errmsg(sqlite3_db_handle(ss.ins_doc)));
+ }
+}
+
+// Insert chunk
+static void sqlite_insert_chunk(SqliteStmts& ss,
+ const std::string& chunk_id,
+ const std::string& doc_id,
+ int source_id,
+ int chunk_index,
+ const std::string& title,
+ const std::string& body,
+ const std::string& meta_json) {
+ sqlite3_reset(ss.ins_chunk);
+ sqlite3_clear_bindings(ss.ins_chunk);
+
+ sqlite_bind_text(ss.ins_chunk, 1, chunk_id);
+ sqlite_bind_text(ss.ins_chunk, 2, doc_id);
+ sqlite3_bind_int(ss.ins_chunk, 3, source_id);
+ sqlite3_bind_int(ss.ins_chunk, 4, chunk_index);
+ sqlite_bind_text(ss.ins_chunk, 5, title);
+ sqlite_bind_text(ss.ins_chunk, 6, body);
+ sqlite_bind_text(ss.ins_chunk, 7, meta_json);
+
+ int rc = sqlite3_step(ss.ins_chunk);
+ if (rc != SQLITE_DONE) {
+ fatal(std::string("SQLite insert rag_chunks failed: ") + sqlite3_errmsg(sqlite3_db_handle(ss.ins_chunk)));
+ }
+}
+
+// Insert into FTS
+static void sqlite_insert_fts(SqliteStmts& ss,
+ const std::string& chunk_id,
+ const std::string& title,
+ const std::string& body) {
+ sqlite3_reset(ss.ins_fts);
+ sqlite3_clear_bindings(ss.ins_fts);
+
+ sqlite_bind_text(ss.ins_fts, 1, chunk_id);
+ sqlite_bind_text(ss.ins_fts, 2, title);
+ sqlite_bind_text(ss.ins_fts, 3, body);
+
+ int rc = sqlite3_step(ss.ins_fts);
+ if (rc != SQLITE_DONE) {
+ fatal(std::string("SQLite insert rag_fts_chunks failed: ") + sqlite3_errmsg(sqlite3_db_handle(ss.ins_fts)));
+ }
+}
+
+// Insert vector row (sqlite3-vec)
+// Schema: rag_vec_chunks(embedding, chunk_id, doc_id, source_id, updated_at)
+static void sqlite_insert_vec(SqliteStmts& ss,
+ const std::vector& emb,
+ const std::string& chunk_id,
+ const std::string& doc_id,
+ int source_id,
+ std::int64_t updated_at_unixepoch) {
+ if (!ss.ins_vec) return;
+
+ sqlite3_reset(ss.ins_vec);
+ sqlite3_clear_bindings(ss.ins_vec);
+
+ bind_vec_embedding(ss.ins_vec, 1, emb);
+ sqlite_bind_text(ss.ins_vec, 2, chunk_id);
+ sqlite_bind_text(ss.ins_vec, 3, doc_id);
+ sqlite3_bind_int(ss.ins_vec, 4, source_id);
+ sqlite3_bind_int64(ss.ins_vec, 5, (sqlite3_int64)updated_at_unixepoch);
+
+ int rc = sqlite3_step(ss.ins_vec);
+ if (rc != SQLITE_DONE) {
+ // In practice, sqlite3-vec may return errors if binding format is wrong.
+ // Keep the message loud and actionable.
+ fatal(std::string("SQLite insert rag_vec_chunks failed (check vec binding format): ")
+ + sqlite3_errmsg(sqlite3_db_handle(ss.ins_vec)));
+ }
+}
+
+// -------------------------
+// Embedding stub
+// -------------------------
+// This function is a placeholder. It returns a deterministic pseudo-embedding from the text.
+// Replace it with a real embedding model call in ProxySQL later.
+//
+// Why deterministic?
+// - Helps test end-to-end ingestion + vector SQL without needing an ML runtime.
+// - Keeps behavior stable across runs.
+//
+static std::vector pseudo_embedding(const std::string& text, int dim) {
+ std::vector v;
+ v.resize((size_t)dim, 0.0f);
+
+ // Simple rolling hash-like accumulation into float bins.
+ // NOT a semantic embedding; only for wiring/testing.
+ std::uint64_t h = 1469598103934665603ULL;
+ for (size_t i = 0; i < text.size(); i++) {
+ h ^= (unsigned char)text[i];
+ h *= 1099511628211ULL;
+
+ // Spread influence into bins
+ size_t idx = (size_t)(h % (std::uint64_t)dim);
+ float val = (float)((h >> 32) & 0xFFFF) / 65535.0f; // 0..1
+ v[idx] += (val - 0.5f);
+ }
+
+ // Very rough normalization
+ double norm = 0.0;
+ for (int i = 0; i < dim; i++) norm += (double)v[(size_t)i] * (double)v[(size_t)i];
+ norm = std::sqrt(norm);
+ if (norm > 1e-12) {
+ for (int i = 0; i < dim; i++) v[(size_t)i] = (float)(v[(size_t)i] / norm);
+ }
+ return v;
+}
+
+// -------------------------
+// Load rag_sources from SQLite
+// -------------------------
+
+static std::vector load_sources(sqlite3* db) {
+ std::vector out;
+
+ const char* sql =
+ "SELECT source_id, name, enabled, "
+ "backend_type, backend_host, backend_port, backend_user, backend_pass, backend_db, "
+ "table_name, pk_column, COALESCE(where_sql,''), "
+ "doc_map_json, chunking_json, COALESCE(embedding_json,'') "
+ "FROM rag_sources WHERE enabled = 1";
+
+ sqlite3_stmt* st = nullptr;
+ sqlite_prepare_or_die(db, &st, sql);
+
+ while (sqlite3_step(st) == SQLITE_ROW) {
+ RagSource s;
+ s.source_id = sqlite3_column_int(st, 0);
+ s.name = (const char*)sqlite3_column_text(st, 1);
+ s.enabled = sqlite3_column_int(st, 2);
+
+ s.backend_type = (const char*)sqlite3_column_text(st, 3);
+ s.host = (const char*)sqlite3_column_text(st, 4);
+ s.port = sqlite3_column_int(st, 5);
+ s.user = (const char*)sqlite3_column_text(st, 6);
+ s.pass = (const char*)sqlite3_column_text(st, 7);
+ s.db = (const char*)sqlite3_column_text(st, 8);
+
+ s.table_name = (const char*)sqlite3_column_text(st, 9);
+ s.pk_column = (const char*)sqlite3_column_text(st, 10);
+ s.where_sql = (const char*)sqlite3_column_text(st, 11);
+
+ const char* doc_map = (const char*)sqlite3_column_text(st, 12);
+ const char* chunk_j = (const char*)sqlite3_column_text(st, 13);
+ const char* emb_j = (const char*)sqlite3_column_text(st, 14);
+
+ try {
+ s.doc_map_json = json::parse(doc_map ? doc_map : "{}");
+ s.chunking_json = json::parse(chunk_j ? chunk_j : "{}");
+ if (emb_j && std::strlen(emb_j) > 0) s.embedding_json = json::parse(emb_j);
+ else s.embedding_json = json(); // null
+ } catch (const std::exception& e) {
+ sqlite3_finalize(st);
+ fatal("Invalid JSON in rag_sources.source_id=" + std::to_string(s.source_id) + ": " + e.what());
+ }
+
+ // Basic validation (fail fast)
+ if (!s.doc_map_json.is_object()) {
+ sqlite3_finalize(st);
+ fatal("doc_map_json must be a JSON object for source_id=" + std::to_string(s.source_id));
+ }
+ if (!s.chunking_json.is_object()) {
+ sqlite3_finalize(st);
+ fatal("chunking_json must be a JSON object for source_id=" + std::to_string(s.source_id));
+ }
+
+ out.push_back(std::move(s));
+ }
+
+ sqlite3_finalize(st);
+ return out;
+}
+
+// -------------------------
+// Build a canonical document from a source row
+// -------------------------
+
+struct BuiltDoc {
+ std::string doc_id;
+ std::string pk_json;
+ std::string title;
+ std::string body;
+ std::string metadata_json;
+};
+
+static BuiltDoc build_document_from_row(const RagSource& src, const RowMap& row) {
+ BuiltDoc d;
+
+ // doc_id
+ if (src.doc_map_json.contains("doc_id") && src.doc_map_json["doc_id"].is_object()
+ && src.doc_map_json["doc_id"].contains("format") && src.doc_map_json["doc_id"]["format"].is_string()) {
+ d.doc_id = apply_format(src.doc_map_json["doc_id"]["format"].get(), row);
+ } else {
+ // fallback: table:pk
+ auto pk = row_get(row, src.pk_column).value_or("");
+ d.doc_id = src.table_name + ":" + pk;
+ }
+
+ // pk_json (refetch pointer)
+ json pk = json::object();
+ pk[src.pk_column] = row_get(row, src.pk_column).value_or("");
+ d.pk_json = json_dump_compact(pk);
+
+ // title/body
+ if (src.doc_map_json.contains("title") && src.doc_map_json["title"].is_object()
+ && src.doc_map_json["title"].contains("concat")) {
+ d.title = eval_concat(src.doc_map_json["title"]["concat"], row, "", false);
+ } else {
+ d.title = "";
+ }
+
+ if (src.doc_map_json.contains("body") && src.doc_map_json["body"].is_object()
+ && src.doc_map_json["body"].contains("concat")) {
+ d.body = eval_concat(src.doc_map_json["body"]["concat"], row, "", false);
+ } else {
+ d.body = "";
+ }
+
+ // metadata_json
+ json meta = json::object();
+ if (src.doc_map_json.contains("metadata")) {
+ meta = build_metadata(src.doc_map_json["metadata"], row);
+ }
+ d.metadata_json = json_dump_compact(meta);
+
+ return d;
+}
+
+// -------------------------
+// Embedding input builder (optional)
+// -------------------------
+
+static std::string build_embedding_input(const EmbeddingConfig& ecfg,
+ const RowMap& row,
+ const std::string& chunk_body) {
+ if (!ecfg.enabled) return "";
+ if (!ecfg.input_spec.is_object()) return chunk_body;
+
+ if (ecfg.input_spec.contains("concat") && ecfg.input_spec["concat"].is_array()) {
+ return eval_concat(ecfg.input_spec["concat"], row, chunk_body, true);
+ }
+
+ return chunk_body;
+}
+
+// -------------------------
+// Ingest one source
+// -------------------------
+
+static SqliteStmts prepare_sqlite_statements(sqlite3* db, bool want_vec) {
+ SqliteStmts ss;
+
+ // Existence check
+ sqlite_prepare_or_die(db, &ss.doc_exists,
+ "SELECT 1 FROM rag_documents WHERE doc_id = ? LIMIT 1");
+
+ // Insert document (v0: no upsert)
+ sqlite_prepare_or_die(db, &ss.ins_doc,
+ "INSERT INTO rag_documents(doc_id, source_id, source_name, pk_json, title, body, metadata_json) "
+ "VALUES(?,?,?,?,?,?,?)");
+
+ // Insert chunk
+ sqlite_prepare_or_die(db, &ss.ins_chunk,
+ "INSERT INTO rag_chunks(chunk_id, doc_id, source_id, chunk_index, title, body, metadata_json) "
+ "VALUES(?,?,?,?,?,?,?)");
+
+ // Insert FTS
+ sqlite_prepare_or_die(db, &ss.ins_fts,
+ "INSERT INTO rag_fts_chunks(chunk_id, title, body) VALUES(?,?,?)");
+
+ // Insert vector (optional)
+ if (want_vec) {
+ // NOTE: If your sqlite3-vec build expects different binding format, adapt bind_vec_embedding().
+ sqlite_prepare_or_die(db, &ss.ins_vec,
+ "INSERT INTO rag_vec_chunks(embedding, chunk_id, doc_id, source_id, updated_at) "
+ "VALUES(?,?,?,?,?)");
+ }
+
+ return ss;
+}
+
+static void ingest_source(sqlite3* sdb, const RagSource& src) {
+ std::cerr << "Ingesting source_id=" << src.source_id
+ << " name=" << src.name
+ << " backend=" << src.backend_type
+ << " table=" << src.table_name << "\n";
+
+ if (src.backend_type != "mysql") {
+ std::cerr << " Skipping: backend_type not supported in v0.\n";
+ return;
+ }
+
+ // Parse chunking & embedding config
+ ChunkingConfig ccfg = parse_chunking_json(src.chunking_json);
+ EmbeddingConfig ecfg = parse_embedding_json(src.embedding_json);
+
+ // Prepare SQLite statements for this run
+ SqliteStmts ss = prepare_sqlite_statements(sdb, ecfg.enabled);
+
+ // Connect MySQL
+ MYSQL* mdb = mysql_connect_or_die(src);
+
+ // Build SELECT
+ std::vector cols = collect_needed_columns(src, ecfg);
+ std::string sel = build_select_sql(src, cols);
+
+ if (mysql_query(mdb, sel.c_str()) != 0) {
+ std::string err = mysql_error(mdb);
+ mysql_close(mdb);
+ sqlite_finalize_all(ss);
+ fatal("MySQL query failed: " + err + "\nSQL: " + sel);
+ }
+
+ MYSQL_RES* res = mysql_store_result(mdb);
+ if (!res) {
+ std::string err = mysql_error(mdb);
+ mysql_close(mdb);
+ sqlite_finalize_all(ss);
+ fatal("mysql_store_result failed: " + err);
+ }
+
+ std::uint64_t ingested_docs = 0;
+ std::uint64_t skipped_docs = 0;
+
+ MYSQL_ROW r;
+ while ((r = mysql_fetch_row(res)) != nullptr) {
+ RowMap row = mysql_row_to_map(res, r);
+
+ BuiltDoc doc = build_document_from_row(src, row);
+
+ // v0: skip if exists
+ if (sqlite_doc_exists(ss, doc.doc_id)) {
+ skipped_docs++;
+ continue;
+ }
+
+ // Insert document
+ sqlite_insert_doc(ss, src.source_id, src.name,
+ doc.doc_id, doc.pk_json, doc.title, doc.body, doc.metadata_json);
+
+ // Chunk and insert chunks + FTS (+ optional vec)
+ std::vector chunks = chunk_text_chars(doc.body, ccfg);
+
+ // Use SQLite's unixepoch() for updated_at normally; vec table also stores updated_at as unix epoch.
+ // Here we store a best-effort "now" from SQLite (unixepoch()) would require a query; instead store 0
+ // or a local time. For v0, we store 0 and let schema default handle other tables.
+ // If you want accuracy, query SELECT unixepoch() once per run and reuse it.
+ std::int64_t now_epoch = 0;
+
+ for (size_t i = 0; i < chunks.size(); i++) {
+ std::string chunk_id = doc.doc_id + "#" + std::to_string(i);
+
+ // Chunk metadata (minimal)
+ json cmeta = json::object();
+ cmeta["chunk_index"] = (int)i;
+
+ std::string chunk_title = doc.title; // simple: repeat doc title
+
+ sqlite_insert_chunk(ss, chunk_id, doc.doc_id, src.source_id, (int)i,
+ chunk_title, chunks[i], json_dump_compact(cmeta));
+
+ sqlite_insert_fts(ss, chunk_id, chunk_title, chunks[i]);
+
+ // Optional vectors
+ if (ecfg.enabled) {
+ // Build embedding input text, then generate pseudo embedding.
+ // Replace pseudo_embedding() with a real embedding provider in ProxySQL.
+ std::string emb_input = build_embedding_input(ecfg, row, chunks[i]);
+ std::vector emb = pseudo_embedding(emb_input, ecfg.dim);
+
+ // Insert into sqlite3-vec table
+ sqlite_insert_vec(ss, emb, chunk_id, doc.doc_id, src.source_id, now_epoch);
+ }
+ }
+
+ ingested_docs++;
+ if (ingested_docs % 1000 == 0) {
+ std::cerr << " progress: ingested_docs=" << ingested_docs
+ << " skipped_docs=" << skipped_docs << "\n";
+ }
+ }
+
+ mysql_free_result(res);
+ mysql_close(mdb);
+ sqlite_finalize_all(ss);
+
+ std::cerr << "Done source " << src.name
+ << " ingested_docs=" << ingested_docs
+ << " skipped_docs=" << skipped_docs << "\n";
+}
+
+// -------------------------
+// Main
+// -------------------------
+
+int main(int argc, char** argv) {
+ if (argc != 2) {
+ std::cerr << "Usage: " << argv[0] << " \n";
+ return 2;
+ }
+
+ const char* sqlite_path = argv[1];
+
+ sqlite3* db = nullptr;
+ if (sqlite3_open(sqlite_path, &db) != SQLITE_OK) {
+ fatal("Could not open SQLite DB: " + std::string(sqlite_path));
+ }
+
+ // Pragmas (safe defaults)
+ sqlite_exec(db, "PRAGMA foreign_keys = ON;");
+ sqlite_exec(db, "PRAGMA journal_mode = WAL;");
+ sqlite_exec(db, "PRAGMA synchronous = NORMAL;");
+
+ // Single transaction for speed
+ if (sqlite_exec(db, "BEGIN IMMEDIATE;") != SQLITE_OK) {
+ sqlite3_close(db);
+ fatal("Failed to begin transaction");
+ }
+
+ bool ok = true;
+ try {
+ std::vector sources = load_sources(db);
+ if (sources.empty()) {
+ std::cerr << "No enabled sources found in rag_sources.\n";
+ }
+ for (size_t i = 0; i < sources.size(); i++) {
+ ingest_source(db, sources[i]);
+ }
+ } catch (const std::exception& e) {
+ std::cerr << "Exception: " << e.what() << "\n";
+ ok = false;
+ } catch (...) {
+ std::cerr << "Unknown exception\n";
+ ok = false;
+ }
+
+ if (ok) {
+ if (sqlite_exec(db, "COMMIT;") != SQLITE_OK) {
+ sqlite_exec(db, "ROLLBACK;");
+ sqlite3_close(db);
+ fatal("Failed to commit transaction");
+ }
+ } else {
+ sqlite_exec(db, "ROLLBACK;");
+ sqlite3_close(db);
+ return 1;
+ }
+
+ sqlite3_close(db);
+ return 0;
+}
+
diff --git a/RAG_POC/schema.sql b/RAG_POC/schema.sql
new file mode 100644
index 0000000000..2a40c3e7a1
--- /dev/null
+++ b/RAG_POC/schema.sql
@@ -0,0 +1,172 @@
+-- ============================================================
+-- ProxySQL RAG Index Schema (SQLite)
+-- v0: documents + chunks + FTS5 + sqlite3-vec embeddings
+-- ============================================================
+
+PRAGMA foreign_keys = ON;
+PRAGMA journal_mode = WAL;
+PRAGMA synchronous = NORMAL;
+
+-- ============================================================
+-- 1) rag_sources: control plane
+-- Defines where to fetch from + how to transform + chunking.
+-- ============================================================
+CREATE TABLE IF NOT EXISTS rag_sources (
+ source_id INTEGER PRIMARY KEY,
+ name TEXT NOT NULL UNIQUE, -- e.g. "stack_posts"
+ enabled INTEGER NOT NULL DEFAULT 1,
+
+ -- Where to retrieve from (PoC: connect directly; later can be "via ProxySQL")
+ backend_type TEXT NOT NULL, -- "mysql" | "postgres" | ...
+ backend_host TEXT NOT NULL,
+ backend_port INTEGER NOT NULL,
+ backend_user TEXT NOT NULL,
+ backend_pass TEXT NOT NULL,
+ backend_db TEXT NOT NULL, -- database/schema name
+
+ table_name TEXT NOT NULL, -- e.g. "posts"
+ pk_column TEXT NOT NULL, -- e.g. "Id"
+
+ -- Optional: restrict ingestion; appended to SELECT as WHERE
+ where_sql TEXT, -- e.g. "PostTypeId IN (1,2)"
+
+ -- REQUIRED: mapping from source row -> rag_documents fields
+ -- JSON spec describing doc_id, title/body concat, metadata pick/rename, etc.
+ doc_map_json TEXT NOT NULL,
+
+ -- REQUIRED: chunking strategy (enabled, chunk_size, overlap, etc.)
+ chunking_json TEXT NOT NULL,
+
+ -- Optional: embedding strategy (how to build embedding input text)
+ -- In v0 you can keep it NULL/empty; define later without schema changes.
+ embedding_json TEXT,
+
+ created_at INTEGER NOT NULL DEFAULT (unixepoch()),
+ updated_at INTEGER NOT NULL DEFAULT (unixepoch())
+);
+
+CREATE INDEX IF NOT EXISTS idx_rag_sources_enabled
+ ON rag_sources(enabled);
+
+CREATE INDEX IF NOT EXISTS idx_rag_sources_backend
+ ON rag_sources(backend_type, backend_host, backend_port, backend_db, table_name);
+
+
+-- ============================================================
+-- 2) rag_documents: canonical documents
+-- One document per source row (e.g. one per posts.Id).
+-- ============================================================
+CREATE TABLE IF NOT EXISTS rag_documents (
+ doc_id TEXT PRIMARY KEY, -- stable: e.g. "posts:12345"
+ source_id INTEGER NOT NULL REFERENCES rag_sources(source_id),
+ source_name TEXT NOT NULL, -- copy of rag_sources.name for convenience
+ pk_json TEXT NOT NULL, -- e.g. {"Id":12345}
+
+ title TEXT,
+ body TEXT,
+ metadata_json TEXT NOT NULL DEFAULT '{}', -- JSON object
+
+ updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+ deleted INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE INDEX IF NOT EXISTS idx_rag_documents_source_updated
+ ON rag_documents(source_id, updated_at);
+
+CREATE INDEX IF NOT EXISTS idx_rag_documents_source_deleted
+ ON rag_documents(source_id, deleted);
+
+
+-- ============================================================
+-- 3) rag_chunks: chunked content
+-- The unit we index in FTS and vectors.
+-- ============================================================
+CREATE TABLE IF NOT EXISTS rag_chunks (
+ chunk_id TEXT PRIMARY KEY, -- e.g. "posts:12345#0"
+ doc_id TEXT NOT NULL REFERENCES rag_documents(doc_id),
+ source_id INTEGER NOT NULL REFERENCES rag_sources(source_id),
+
+ chunk_index INTEGER NOT NULL, -- 0..N-1
+ title TEXT,
+ body TEXT NOT NULL,
+
+ -- Optional per-chunk metadata (e.g. offsets, has_code, section label)
+ metadata_json TEXT NOT NULL DEFAULT '{}',
+
+ updated_at INTEGER NOT NULL DEFAULT (unixepoch()),
+ deleted INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS uq_rag_chunks_doc_idx
+ ON rag_chunks(doc_id, chunk_index);
+
+CREATE INDEX IF NOT EXISTS idx_rag_chunks_source_doc
+ ON rag_chunks(source_id, doc_id);
+
+CREATE INDEX IF NOT EXISTS idx_rag_chunks_deleted
+ ON rag_chunks(deleted);
+
+
+-- ============================================================
+-- 4) rag_fts_chunks: FTS5 index (contentless)
+-- Maintained explicitly by the ingester.
+-- Notes:
+-- - chunk_id is stored but UNINDEXED.
+-- - Use bm25(rag_fts_chunks) for ranking.
+-- ============================================================
+CREATE VIRTUAL TABLE IF NOT EXISTS rag_fts_chunks
+USING fts5(
+ chunk_id UNINDEXED,
+ title,
+ body,
+ tokenize = 'unicode61'
+);
+
+
+-- ============================================================
+-- 5) rag_vec_chunks: sqlite3-vec index
+-- Stores embeddings per chunk for vector search.
+--
+-- IMPORTANT:
+-- - dimension must match your embedding model (example: 1536).
+-- - metadata columns are included to help join/filter.
+-- ============================================================
+CREATE VIRTUAL TABLE IF NOT EXISTS rag_vec_chunks
+USING vec0(
+ embedding float[1536], -- change if you use another dimension
+ chunk_id TEXT, -- join key back to rag_chunks
+ doc_id TEXT, -- optional convenience
+ source_id INTEGER, -- optional convenience
+ updated_at INTEGER -- optional convenience
+);
+
+-- Optional: convenience view for debugging / SQL access patterns
+CREATE VIEW IF NOT EXISTS rag_chunk_view AS
+SELECT
+ c.chunk_id,
+ c.doc_id,
+ c.source_id,
+ d.source_name,
+ d.pk_json,
+ COALESCE(c.title, d.title) AS title,
+ c.body,
+ d.metadata_json AS doc_metadata_json,
+ c.metadata_json AS chunk_metadata_json,
+ c.updated_at
+FROM rag_chunks c
+JOIN rag_documents d ON d.doc_id = c.doc_id
+WHERE c.deleted = 0 AND d.deleted = 0;
+
+
+-- ============================================================
+-- 6) (Optional) sync state placeholder for later incremental ingestion
+-- Not used in v0, but reserving it avoids later schema churn.
+-- ============================================================
+CREATE TABLE IF NOT EXISTS rag_sync_state (
+ source_id INTEGER PRIMARY KEY REFERENCES rag_sources(source_id),
+ mode TEXT NOT NULL DEFAULT 'poll', -- 'poll' | 'cdc'
+ cursor_json TEXT NOT NULL DEFAULT '{}', -- watermark/checkpoint
+ last_ok_at INTEGER,
+ last_error TEXT
+);
+
diff --git a/RAG_POC/sql-examples.md b/RAG_POC/sql-examples.md
new file mode 100644
index 0000000000..b7b52128f4
--- /dev/null
+++ b/RAG_POC/sql-examples.md
@@ -0,0 +1,348 @@
+# ProxySQL RAG Index — SQL Examples (FTS, Vectors, Hybrid)
+
+This file provides concrete SQL examples for querying the ProxySQL-hosted SQLite RAG index directly (for debugging, internal dashboards, or SQL-native applications).
+
+The **preferred interface for AI agents** remains MCP tools (`mcp-tools.md`). SQL access should typically be restricted to trusted callers.
+
+Assumed tables:
+- `rag_documents`
+- `rag_chunks`
+- `rag_fts_chunks` (FTS5)
+- `rag_vec_chunks` (sqlite3-vec vec0 table)
+
+---
+
+## 0. Common joins and inspection
+
+### 0.1 Inspect one document and its chunks
+```sql
+SELECT * FROM rag_documents WHERE doc_id = 'posts:12345';
+SELECT * FROM rag_chunks WHERE doc_id = 'posts:12345' ORDER BY chunk_index;
+```
+
+### 0.2 Use the convenience view (if enabled)
+```sql
+SELECT * FROM rag_chunk_view WHERE doc_id = 'posts:12345' ORDER BY chunk_id;
+```
+
+---
+
+## 1. FTS5 examples
+
+### 1.1 Basic FTS search (top 10)
+```sql
+SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts_raw
+FROM rag_fts_chunks f
+WHERE rag_fts_chunks MATCH 'json_extract mysql'
+ORDER BY score_fts_raw
+LIMIT 10;
+```
+
+### 1.2 Join FTS results to chunk text and document metadata
+```sql
+SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts_raw,
+ c.doc_id,
+ COALESCE(c.title, d.title) AS title,
+ c.body AS chunk_body,
+ d.metadata_json AS doc_metadata_json
+FROM rag_fts_chunks f
+JOIN rag_chunks c ON c.chunk_id = f.chunk_id
+JOIN rag_documents d ON d.doc_id = c.doc_id
+WHERE rag_fts_chunks MATCH 'json_extract mysql'
+ AND c.deleted = 0 AND d.deleted = 0
+ORDER BY score_fts_raw
+LIMIT 10;
+```
+
+### 1.3 Apply a source filter (by source_id)
+```sql
+SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts_raw
+FROM rag_fts_chunks f
+JOIN rag_chunks c ON c.chunk_id = f.chunk_id
+WHERE rag_fts_chunks MATCH 'replication lag'
+ AND c.source_id = 1
+ORDER BY score_fts_raw
+LIMIT 20;
+```
+
+### 1.4 Phrase queries, boolean operators (FTS5)
+```sql
+-- phrase
+SELECT chunk_id FROM rag_fts_chunks
+WHERE rag_fts_chunks MATCH '"group replication"'
+LIMIT 20;
+
+-- boolean: term1 AND term2
+SELECT chunk_id FROM rag_fts_chunks
+WHERE rag_fts_chunks MATCH 'mysql AND deadlock'
+LIMIT 20;
+
+-- boolean: term1 NOT term2
+SELECT chunk_id FROM rag_fts_chunks
+WHERE rag_fts_chunks MATCH 'mysql NOT mariadb'
+LIMIT 20;
+```
+
+---
+
+## 2. Vector search examples (sqlite3-vec)
+
+Vector SQL varies slightly depending on sqlite3-vec build and how you bind vectors.
+Below are **two patterns** you can implement in ProxySQL.
+
+### 2.1 Pattern A (recommended): ProxySQL computes embeddings; SQL receives a bound vector
+In this pattern, ProxySQL:
+1) Computes the query embedding in C++
+2) Executes SQL with a bound parameter `:qvec` representing the embedding
+
+A typical “nearest neighbors” query shape is:
+
+```sql
+-- PSEUDOCODE: adapt to sqlite3-vec's exact operator/function in your build.
+SELECT
+ v.chunk_id,
+ v.distance AS distance_raw
+FROM rag_vec_chunks v
+WHERE v.embedding MATCH :qvec
+ORDER BY distance_raw
+LIMIT 10;
+```
+
+Then join to chunks:
+```sql
+-- PSEUDOCODE: join with content and metadata
+SELECT
+ v.chunk_id,
+ v.distance AS distance_raw,
+ c.doc_id,
+ c.body AS chunk_body,
+ d.metadata_json AS doc_metadata_json
+FROM (
+ SELECT chunk_id, distance
+ FROM rag_vec_chunks
+ WHERE embedding MATCH :qvec
+ ORDER BY distance
+ LIMIT 10
+) v
+JOIN rag_chunks c ON c.chunk_id = v.chunk_id
+JOIN rag_documents d ON d.doc_id = c.doc_id;
+```
+
+### 2.2 Pattern B (debug): store a query vector in a temporary table
+This is useful when you want to run vector queries manually in SQL without MCP support.
+
+```sql
+CREATE TEMP TABLE tmp_query_vec(qvec BLOB);
+-- Insert the query vector (float32 array blob). The insertion is usually done by tooling, not manually.
+-- INSERT INTO tmp_query_vec VALUES (X'...');
+
+-- PSEUDOCODE: use tmp_query_vec.qvec as the query embedding
+SELECT
+ v.chunk_id,
+ v.distance
+FROM rag_vec_chunks v, tmp_query_vec t
+WHERE v.embedding MATCH t.qvec
+ORDER BY v.distance
+LIMIT 10;
+```
+
+---
+
+## 3. Hybrid search examples
+
+Hybrid retrieval is best implemented in the MCP layer because it mixes ranking systems and needs careful bounding.
+However, you can approximate hybrid behavior using SQL to validate logic.
+
+### 3.1 Hybrid Mode A: Parallel FTS + Vector then fuse (RRF)
+
+#### Step 1: FTS top 50 (ranked)
+```sql
+WITH fts AS (
+ SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts_raw
+ FROM rag_fts_chunks f
+ WHERE rag_fts_chunks MATCH :fts_query
+ ORDER BY score_fts_raw
+ LIMIT 50
+)
+SELECT * FROM fts;
+```
+
+#### Step 2: Vector top 50 (ranked)
+```sql
+WITH vec AS (
+ SELECT
+ v.chunk_id,
+ v.distance AS distance_raw
+ FROM rag_vec_chunks v
+ WHERE v.embedding MATCH :qvec
+ ORDER BY v.distance
+ LIMIT 50
+)
+SELECT * FROM vec;
+```
+
+#### Step 3: Fuse via Reciprocal Rank Fusion (RRF)
+In SQL you need ranks. SQLite supports window functions in modern builds.
+
+```sql
+WITH
+fts AS (
+ SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts_raw,
+ ROW_NUMBER() OVER (ORDER BY bm25(rag_fts_chunks)) AS rank_fts
+ FROM rag_fts_chunks f
+ WHERE rag_fts_chunks MATCH :fts_query
+ LIMIT 50
+),
+vec AS (
+ SELECT
+ v.chunk_id,
+ v.distance AS distance_raw,
+ ROW_NUMBER() OVER (ORDER BY v.distance) AS rank_vec
+ FROM rag_vec_chunks v
+ WHERE v.embedding MATCH :qvec
+ LIMIT 50
+),
+merged AS (
+ SELECT
+ COALESCE(fts.chunk_id, vec.chunk_id) AS chunk_id,
+ fts.rank_fts,
+ vec.rank_vec,
+ fts.score_fts_raw,
+ vec.distance_raw
+ FROM fts
+ FULL OUTER JOIN vec ON vec.chunk_id = fts.chunk_id
+),
+rrf AS (
+ SELECT
+ chunk_id,
+ score_fts_raw,
+ distance_raw,
+ rank_fts,
+ rank_vec,
+ (1.0 / (60.0 + COALESCE(rank_fts, 1000000))) +
+ (1.0 / (60.0 + COALESCE(rank_vec, 1000000))) AS score_rrf
+ FROM merged
+)
+SELECT
+ r.chunk_id,
+ r.score_rrf,
+ c.doc_id,
+ c.body AS chunk_body
+FROM rrf r
+JOIN rag_chunks c ON c.chunk_id = r.chunk_id
+ORDER BY r.score_rrf DESC
+LIMIT 10;
+```
+
+**Important**: SQLite does not support `FULL OUTER JOIN` directly in all builds.
+For production, implement the merge/fuse in C++ (MCP layer). This SQL is illustrative.
+
+### 3.2 Hybrid Mode B: Broad FTS then vector rerank (candidate generation)
+
+#### Step 1: FTS candidate set (top 200)
+```sql
+WITH candidates AS (
+ SELECT
+ f.chunk_id,
+ bm25(rag_fts_chunks) AS score_fts_raw
+ FROM rag_fts_chunks f
+ WHERE rag_fts_chunks MATCH :fts_query
+ ORDER BY score_fts_raw
+ LIMIT 200
+)
+SELECT * FROM candidates;
+```
+
+#### Step 2: Vector rerank within candidates
+Conceptually:
+- Join candidates to `rag_vec_chunks` and compute distance to `:qvec`
+- Keep top 10
+
+```sql
+WITH candidates AS (
+ SELECT
+ f.chunk_id
+ FROM rag_fts_chunks f
+ WHERE rag_fts_chunks MATCH :fts_query
+ ORDER BY bm25(rag_fts_chunks)
+ LIMIT 200
+),
+reranked AS (
+ SELECT
+ v.chunk_id,
+ v.distance AS distance_raw
+ FROM rag_vec_chunks v
+ JOIN candidates c ON c.chunk_id = v.chunk_id
+ WHERE v.embedding MATCH :qvec
+ ORDER BY v.distance
+ LIMIT 10
+)
+SELECT
+ r.chunk_id,
+ r.distance_raw,
+ ch.doc_id,
+ ch.body
+FROM reranked r
+JOIN rag_chunks ch ON ch.chunk_id = r.chunk_id;
+```
+
+As above, the exact `MATCH :qvec` syntax may need adaptation to your sqlite3-vec build; implement vector query execution in C++ and keep SQL as internal glue.
+
+---
+
+## 4. Common “application-friendly” queries
+
+### 4.1 Return doc_id + score + title only (no bodies)
+```sql
+SELECT
+ f.chunk_id,
+ c.doc_id,
+ COALESCE(c.title, d.title) AS title,
+ bm25(rag_fts_chunks) AS score_fts_raw
+FROM rag_fts_chunks f
+JOIN rag_chunks c ON c.chunk_id = f.chunk_id
+JOIN rag_documents d ON d.doc_id = c.doc_id
+WHERE rag_fts_chunks MATCH :q
+ORDER BY score_fts_raw
+LIMIT 20;
+```
+
+### 4.2 Return top doc_ids (deduplicate by doc_id)
+```sql
+WITH ranked_chunks AS (
+ SELECT
+ c.doc_id,
+ bm25(rag_fts_chunks) AS score_fts_raw
+ FROM rag_fts_chunks f
+ JOIN rag_chunks c ON c.chunk_id = f.chunk_id
+ WHERE rag_fts_chunks MATCH :q
+ ORDER BY score_fts_raw
+ LIMIT 200
+)
+SELECT doc_id, MIN(score_fts_raw) AS best_score
+FROM ranked_chunks
+GROUP BY doc_id
+ORDER BY best_score
+LIMIT 20;
+```
+
+---
+
+## 5. Practical guidance
+
+- Use SQL mode mainly for debugging and internal tooling.
+- Prefer MCP tools for agent interaction:
+ - stable schemas
+ - strong guardrails
+ - consistent hybrid scoring
+- Implement hybrid fusion in C++ (not in SQL) to avoid dialect limitations and to keep scoring correct.
diff --git a/doc/MCP/Architecture.md b/doc/MCP/Architecture.md
index 342db909c7..ad8a0883f4 100644
--- a/doc/MCP/Architecture.md
+++ b/doc/MCP/Architecture.md
@@ -1,6 +1,6 @@
# MCP Architecture
-This document describes the architecture of the MCP (Model Context Protocol) module in ProxySQL, including endpoint design, tool handler implementation, and future architectural direction.
+This document describes the architecture of the MCP (Model Context Protocol) module in ProxySQL, including endpoint design and tool handler implementation.
## Overview
@@ -14,7 +14,7 @@ The MCP module implements JSON-RPC 2.0 over HTTPS for LLM (Large Language Model)
- **Endpoint Authentication**: Per-endpoint Bearer token authentication
- **Connection Pooling**: MySQL connection pooling for efficient database access
-## Current Architecture
+## Implemented Architecture
### Component Diagram
@@ -27,7 +27,12 @@ The MCP module implements JSON-RPC 2.0 over HTTPS for LLM (Large Language Model)
│ │ - Configuration variables (mcp-*) │ │
│ │ - Status variables │ │
│ │ - mcp_server (ProxySQL_MCP_Server) │ │
-│ │ - mysql_tool_handler (MySQL_Tool_Handler) │ │
+│ │ - config_tool_handler (NEW) │ │
+│ │ - query_tool_handler (NEW) │ │
+│ │ - admin_tool_handler (NEW) │ │
+│ │ - cache_tool_handler (NEW) │ │
+│ │ - observe_tool_handler (NEW) │ │
+│ │ - ai_tool_handler (NEW) │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
@@ -39,45 +44,30 @@ The MCP module implements JSON-RPC 2.0 over HTTPS for LLM (Large Language Model)
│ │ SSL: Uses ProxySQL's certificates │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│ │ │
-│ ┌─────────────────────┼─────────────────────┐ │
-│ ▼ ▼ ▼ │
-│ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │
-│ │ /mcp/config │ │ /mcp/observe │ │ /mcp/query │ │
-│ │ MCP_JSONRPC_ │ │ MCP_JSONRPC_ │ │ MCP_JSONRPC_ │ │
-│ │ Resource │ │ Resource │ │ Resource │ │
-│ └─────────┬─────────┘ └─────────┬─────────┘ └─────────┬─────────┘ │
-│ │ │ │ │
-│ └─────────────────────┼─────────────────────┘ │
-│ ▼ │
-│ ┌────────────────────────────────────────────┐ │
-│ │ MySQL_Tool_Handler (Shared) │ │
-│ │ │ │
-│ │ Tools: │ │
-│ │ - list_schemas │ │
-│ │ - list_tables │ │
-│ │ - describe_table │ │
-│ │ - get_constraints │ │
-│ │ - table_profile │ │
-│ │ - column_profile │ │
-│ │ - sample_rows │ │
-│ │ - run_sql_readonly │ │
-│ │ - catalog_* (6 tools) │ │
-│ └────────────────────────────────────────────┘ │
-│ │ │
-│ ▼ │
-│ ┌────────────────────────────────────────────┐ │
-│ │ MySQL Backend │ │
-│ │ (Connection Pool) │ │
-│ └────────────────────────────────────────────┘ │
+│ ┌──────────────┬──────────────┼──────────────┬──────────────┬─────────┐ │
+│ ▼ ▼ ▼ ▼ ▼ ▼ │
+│ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌───┐│
+│ │conf│ │obs │ │qry │ │adm │ │cach│ │ai ││
+│ │TH │ │TH │ │TH │ │TH │ │TH │ │TH ││
+│ └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └─┬─┘│
+│ │ │ │ │ │ │ │
+│ │ │ │ │ │ │ │
+│ Tools: Tools: Tools: Tools: Tools: │ │
+│ - get_config - list_ - list_ - admin_ - get_ │ │
+│ - set_config stats schemas - set_ cache │ │
+│ - reload - show_ - list_ - reload - set_ │ │
+│ metrics tables - invalidate │ │
+│ - query │ │
+│ │ │
+│ ┌────────────────────────────────────────────┐ │
+│ │ MySQL Backend │ │
+│ │ (Connection Pool) │ │
+│ └────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
-### Current Limitations
-
-1. **All endpoints share the same tool handler** - No differentiation between endpoints
-2. **Same tools available everywhere** - No specialized tools per endpoint
-3. **Single connection pool** - All queries use the same MySQL connections
-4. **No per-endpoint authentication in code** - Variables exist but not implemented
+Where:
+- `TH` = Tool Handler
### File Structure
@@ -85,19 +75,33 @@ The MCP module implements JSON-RPC 2.0 over HTTPS for LLM (Large Language Model)
include/
├── MCP_Thread.h # MCP_Threads_Handler class definition
├── MCP_Endpoint.h # MCP_JSONRPC_Resource class definition
-├── MySQL_Tool_Handler.h # MySQL_Tool_Handler class definition
-├── MySQL_Catalog.h # SQLite catalog for LLM memory
+├── MCP_Tool_Handler.h # Base class for all tool handlers
+├── Config_Tool_Handler.h # Configuration endpoint tool handler
+├── Query_Tool_Handler.h # Query endpoint tool handler (includes discovery tools)
+├── Admin_Tool_Handler.h # Administration endpoint tool handler
+├── Cache_Tool_Handler.h # Cache endpoint tool handler
+├── Observe_Tool_Handler.h # Observability endpoint tool handler
+├── AI_Tool_Handler.h # AI endpoint tool handler
+├── Discovery_Schema.h # Discovery catalog implementation
+├── Static_Harvester.h # Static database harvester for discovery
└── ProxySQL_MCP_Server.hpp # ProxySQL_MCP_Server class definition
lib/
├── MCP_Thread.cpp # MCP_Threads_Handler implementation
├── MCP_Endpoint.cpp # MCP_JSONRPC_Resource implementation
-├── MySQL_Tool_Handler.cpp # MySQL_Tool_Handler implementation
-├── MySQL_Catalog.cpp # SQLite catalog implementation
+├── MCP_Tool_Handler.cpp # Base class implementation
+├── Config_Tool_Handler.cpp # Configuration endpoint implementation
+├── Query_Tool_Handler.cpp # Query endpoint implementation
+├── Admin_Tool_Handler.cpp # Administration endpoint implementation
+├── Cache_Tool_Handler.cpp # Cache endpoint implementation
+├── Observe_Tool_Handler.cpp # Observability endpoint implementation
+├── AI_Tool_Handler.cpp # AI endpoint implementation
+├── Discovery_Schema.cpp # Discovery catalog implementation
+├── Static_Harvester.cpp # Static database harvester implementation
└── ProxySQL_MCP_Server.cpp # HTTPS server implementation
```
-### Request Flow (Current)
+### Request Flow (Implemented)
```
1. LLM Client → POST /mcp/{endpoint} → HTTPS Server (port 6071)
@@ -107,67 +111,22 @@ lib/
- initialize/ping → Handled directly
- tools/list → handle_tools_list()
- tools/describe → handle_tools_describe()
- - tools/call → handle_tools_call() → MySQL_Tool_Handler
-5. MySQL_Tool_Handler → MySQL Backend (via connection pool)
+ - tools/call → handle_tools_call() → Dedicated Tool Handler
+5. Dedicated Tool Handler → MySQL Backend (via connection pool)
6. Return JSON-RPC response
```
-## Future Architecture: Multiple Tool Handlers
+## Implemented Endpoint Specifications
-### Goal
+### Overview
-Each MCP endpoint will have its own dedicated tool handler with specific tools designed for that endpoint's purpose. This allows for:
+Each MCP endpoint has its own dedicated tool handler with specific tools designed for that endpoint's purpose. This allows for:
- **Specialized tools** - Different tools for different purposes
- **Isolated resources** - Separate connection pools per endpoint
- **Independent authentication** - Per-endpoint credentials
- **Clear separation of concerns** - Each endpoint has a well-defined purpose
-### Target Architecture
-
-```
-┌─────────────────────────────────────────────────────────────────────────────┐
-│ ProxySQL Process │
-│ │
-│ ┌──────────────────────────────────────────────────────────────────────┐ │
-│ │ MCP_Threads_Handler │ │
-│ │ - Configuration variables │ │
-│ │ - Status variables │ │
-│ │ - mcp_server │ │
-│ │ - config_tool_handler (NEW) │ │
-│ │ - query_tool_handler (NEW) │ │
-│ │ - admin_tool_handler (NEW) │ │
-│ │ - cache_tool_handler (NEW) │ │
-│ │ - observe_tool_handler (NEW) │ │
-│ └──────────────────────────────────────────────────────────────────────┘ │
-│ │ │
-│ ▼ │
-│ ┌──────────────────────────────────────────────────────────────────────┐ │
-│ │ ProxySQL_MCP_Server │ │
-│ │ (Single HTTPS Server) │ │
-│ └──────────────────────────────────────────────────────────────────────┘ │
-│ │ │
-│ ┌──────────────┬──────────────┼──────────────┬──────────────┬─────────┐ │
-│ ▼ ▼ ▼ ▼ ▼ ▼ │
-│ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌───┐│
-│ │conf│ │obs │ │qry │ │adm │ │cach│ │cat││
-│ │TH │ │TH │ │TH │ │TH │ │TH │ │log│││
-│ └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └─┬─┘│
-│ │ │ │ │ │ │ │
-│ │ │ │ │ │ │ │
-│ Tools: Tools: Tools: Tools: Tools: │ │
-│ - get_config - list_ - list_ - admin_ - get_ │ │
-│ - set_config stats schemas - set_ cache │ │
-│ - reload - show_ - list_ - reload - set_ │ │
-│ metrics tables - invalidate │ │
-│ - query │ │
-│ │ │
-└─────────────────────────────────────────────────────────────────────────────┘
-```
-
-Where:
-- `TH` = Tool Handler
-
### Endpoint Specifications
#### `/mcp/config` - Configuration Endpoint
@@ -223,11 +182,26 @@ Where:
- `sample_rows` - Get sample data
- `run_sql_readonly` - Execute read-only SQL
- `explain_sql` - Explain query execution plan
+- `suggest_joins` - Suggest join paths between tables
+- `find_reference_candidates` - Find potential foreign key relationships
+- `table_profile` - Get table statistics and data distribution
+- `column_profile` - Get column statistics and data distribution
+- `sample_distinct` - Get distinct values from a column
+- `catalog_get` - Get entry from discovery catalog
+- `catalog_upsert` - Insert or update entry in discovery catalog
+- `catalog_delete` - Delete entry from discovery catalog
+- `catalog_search` - Search entries in discovery catalog
+- `catalog_list` - List all entries in discovery catalog
+- `catalog_clear` - Clear all entries from discovery catalog
+- `discovery.run_static` - Run static database discovery (Phase 1)
+- `agent.*` - Agent coordination tools for discovery
+- `llm.*` - LLM interaction tools for discovery
**Use Cases**:
- LLM assistants for database exploration
- Data analysis and discovery
- Query optimization assistance
+- Two-phase discovery (static harvest + LLM analysis)
**Authentication**: `mcp-query_endpoint_auth` (Bearer token)
@@ -276,6 +250,25 @@ Where:
---
+#### `/mcp/ai` - AI Endpoint
+
+**Purpose**: AI and LLM features
+
+**Tools**:
+- `llm.query` - Query LLM with database context
+- `llm.analyze` - Analyze data with LLM
+- `llm.generate` - Generate content with LLM
+- `anomaly.detect` - Detect anomalies in data
+- `anomaly.list` - List detected anomalies
+- `recommendation.get` - Get AI recommendations
+
+**Use Cases**:
+- LLM-powered data analysis
+- Anomaly detection
+- AI-driven recommendations
+
+**Authentication**: `mcp-ai_endpoint_auth` (Bearer token)
+
### Tool Discovery Flow
MCP clients should discover available tools dynamically:
@@ -406,51 +399,53 @@ private:
};
```
-## Implementation Roadmap
+## Implementation Status
-### Phase 1: Base Infrastructure
+### Phase 1: Base Infrastructure ✅ COMPLETED
-1. Create `MCP_Tool_Handler` base class
-2. Create stub implementations for all 5 tool handlers
-3. Update `MCP_Threads_Handler` to manage all handlers
-4. Update `ProxySQL_MCP_Server` to pass handlers to endpoints
+1. ✅ Create `MCP_Tool_Handler` base class
+2. ✅ Create implementations for all 6 tool handlers (config, query, admin, cache, observe, ai)
+3. ✅ Update `MCP_Threads_Handler` to manage all handlers
+4. ✅ Update `ProxySQL_MCP_Server` to pass handlers to endpoints
-### Phase 2: Tool Implementation
+### Phase 2: Tool Implementation ✅ COMPLETED
-1. Implement Config_Tool_Handler tools
-2. Implement Query_Tool_Handler tools (move from MySQL_Tool_Handler)
-3. Implement Admin_Tool_Handler tools
-4. Implement Cache_Tool_Handler tools
-5. Implement Observe_Tool_Handler tools
+1. ✅ Implement Config_Tool_Handler tools
+2. ✅ Implement Query_Tool_Handler tools (includes MySQL tools and discovery tools)
+3. ✅ Implement Admin_Tool_Handler tools
+4. ✅ Implement Cache_Tool_Handler tools
+5. ✅ Implement Observe_Tool_Handler tools
+6. ✅ Implement AI_Tool_Handler tools
-### Phase 3: Authentication & Testing
+### Phase 3: Authentication & Testing ✅ MOSTLY COMPLETED
1. ✅ Implement per-endpoint authentication
2. ⚠️ Update test scripts to use dynamic tool discovery
3. ⚠️ Add integration tests for each endpoint
-4. ⚠️ Documentation updates
+4. ✅ Documentation updates (this document)
-## Migration Strategy
+## Migration Status ✅ COMPLETED
-### Backward Compatibility
+### Backward Compatibility Maintained
-The migration to multiple tool handlers will maintain backward compatibility:
+The migration to multiple tool handlers has been completed while maintaining backward compatibility:
-1. The existing `mysql_tool_handler` will be renamed to `query_tool_handler`
-2. Existing tools will continue to work on `/mcp/query`
-3. New endpoints will be added incrementally
-4. Deprecation warnings for accessing tools on wrong endpoints
+1. ✅ The existing `mysql_tool_handler` has been replaced by `query_tool_handler`
+2. ✅ Existing tools continue to work on `/mcp/query`
+3. ✅ New endpoints have been added incrementally
+4. ✅ Deprecation warnings are provided for accessing tools on wrong endpoints
-### Gradual Migration
+### Migration Steps Completed
```
-Step 1: Add new base class and stub handlers (no behavior change)
-Step 2: Implement /mcp/config endpoint (new functionality)
-Step 3: Move MySQL tools to /mcp/query (existing tools migrate)
-Step 4: Implement /mcp/admin (new functionality)
-Step 5: Implement /mcp/cache (new functionality)
-Step 6: Implement /mcp/observe (new functionality)
-Step 7: Enable per-endpoint auth
+✅ Step 1: Add new base class and stub handlers (no behavior change)
+✅ Step 2: Implement /mcp/config endpoint (new functionality)
+✅ Step 3: Move MySQL tools to /mcp/query (existing tools migrate)
+✅ Step 4: Implement /mcp/admin (new functionality)
+✅ Step 5: Implement /mcp/cache (new functionality)
+✅ Step 6: Implement /mcp/observe (new functionality)
+✅ Step 7: Enable per-endpoint auth
+✅ Step 8: Add /mcp/ai endpoint (new AI functionality)
```
## Related Documentation
@@ -462,4 +457,4 @@ Step 7: Enable per-endpoint auth
- **MCP Thread Version:** 0.1.0
- **Architecture Version:** 1.0 (design document)
-- **Last Updated:** 2025-01-12
+- **Last Updated:** 2026-01-19
diff --git a/doc/MCP/Database_Discovery_Agent.md b/doc/MCP/Database_Discovery_Agent.md
index 58eaf01f00..3af3c88a76 100644
--- a/doc/MCP/Database_Discovery_Agent.md
+++ b/doc/MCP/Database_Discovery_Agent.md
@@ -1,8 +1,10 @@
-# Database Discovery Agent Architecture
+# Database Discovery Agent Architecture (Conceptual Design)
## Overview
-This document describes the architecture for an AI-powered database discovery agent that can autonomously explore, understand, and analyze any database schema regardless of complexity or domain. The agent uses a mixture-of-experts approach where specialized LLM agents collaborate to build comprehensive understanding of database structures, data patterns, and business semantics.
+This document describes a conceptual architecture for an AI-powered database discovery agent that could autonomously explore, understand, and analyze any database schema regardless of complexity or domain. The agent would use a mixture-of-experts approach where specialized LLM agents collaborate to build comprehensive understanding of database structures, data patterns, and business semantics.
+
+**Note:** This is a conceptual design document. The actual ProxySQL MCP implementation uses a different approach based on the two-phase discovery architecture described in `Two_Phase_Discovery_Implementation.md`.
## Core Principles
@@ -798,3 +800,12 @@ relationships = agent.catalog.get_kind("relationship")
## Version History
- **1.0** (2025-01-12) - Initial architecture design
+
+## Implementation Status
+
+**Status:** Conceptual design - Not implemented
+**Actual Implementation:** See for the actual ProxySQL MCP discovery implementation.
+
+## Version
+
+- **Last Updated:** 2026-01-19
diff --git a/doc/MCP/FTS_Implementation_Plan.md b/doc/MCP/FTS_Implementation_Plan.md
index 4a06d4aaec..e6062abfc5 100644
--- a/doc/MCP/FTS_Implementation_Plan.md
+++ b/doc/MCP/FTS_Implementation_Plan.md
@@ -1,8 +1,10 @@
-# Full Text Search (FTS) Implementation Plan
+# Full Text Search (FTS) Implementation Status
## Overview
-This document describes the implementation of Full Text Search (FTS) capabilities for the ProxySQL MCP Query endpoint. The FTS system enables AI agents to quickly search indexed data before querying the full MySQL database, using SQLite's FTS5 extension.
+This document describes the current implementation of Full Text Search (FTS) capabilities in ProxySQL MCP. The FTS system enables AI agents to quickly search indexed database metadata and LLM-generated artifacts using SQLite's FTS5 extension.
+
+**Status: IMPLEMENTED** ✅
## Requirements
@@ -21,453 +23,224 @@ MCP Query Endpoint
↓
Query_Tool_Handler (routes tool calls)
↓
-MySQL_Tool_Handler (implements tools)
- ↓
-MySQL_FTS (new class - manages FTS database)
+Discovery_Schema (manages FTS database)
↓
-SQLite FTS5 (mcp_fts.db)
+SQLite FTS5 (mcp_catalog.db)
```
### Database Design
-**Separate SQLite database**: `mcp_fts.db` (configurable via `mcp-ftspath` variable)
-
-**Tables**:
-- `fts_indexes` - Metadata for all indexes
-- `fts_data_` - Content tables (one per index)
-- `fts_search_` - FTS5 virtual tables (one per index)
+**Integrated with Discovery Schema**: FTS functionality is built into the existing `mcp_catalog.db` database.
-## Tools (6 total)
+**FTS Tables**:
+- `fts_objects` - FTS5 index over database objects (contentless)
+- `fts_llm` - FTS5 index over LLM-generated artifacts (with content)
-### 1. fts_index_table
-Create and populate an FTS index for a MySQL table.
+## Tools (Integrated with Discovery Tools)
-**Parameters**:
-| Name | Type | Required | Description |
-|------|------|----------|-------------|
-| schema | string | Yes | Schema name |
-| table | string | Yes | Table name |
-| columns | string | Yes | JSON array of column names to index |
-| primary_key | string | Yes | Primary key column name |
-| where_clause | string | No | Optional WHERE clause for filtering |
+### 1. catalog_search
-**Response**:
-```json
-{
- "success": true,
- "schema": "sales",
- "table": "orders",
- "row_count": 15000,
- "indexed_at": 1736668800
-}
-```
-
-**Implementation Logic**:
-1. Validate parameters (table exists, columns are valid)
-2. Check if index already exists
-3. Create dynamic tables: `fts_data__` and `fts_search__`
-4. Fetch all rows from MySQL using `execute_query()`
-5. For each row:
- - Concatenate indexed column values into searchable content
- - Store original row data as JSON metadata
- - Insert into data table (triggers sync to FTS)
-6. Update `fts_indexes` metadata
-7. Return result
-
-### 2. fts_search
-
-Search indexed data using FTS5.
+Search indexed data using FTS5 across both database objects and LLM artifacts.
**Parameters**:
| Name | Type | Required | Description |
|------|------|----------|-------------|
| query | string | Yes | FTS5 search query |
-| schema | string | No | Filter by schema |
-| table | string | No | Filter by table |
-| limit | integer | No | Max results (default: 100) |
-| offset | integer | No | Pagination offset (default: 0) |
+| include_objects | boolean | No | Include detailed object information (default: false) |
+| object_limit | integer | No | Max objects to return when include_objects=true (default: 50) |
**Response**:
```json
{
"success": true,
- "query": "urgent order",
- "total_matches": 234,
+ "query": "customer order",
"results": [
{
- "schema": "sales",
- "table": "orders",
- "primary_key_value": "12345",
- "snippet": "Customer has urgent order...",
- "metadata": "{\"order_id\":12345,\"customer_id\":987,...}"
- }
- ]
-}
-```
-
-**Implementation Logic**:
-1. Build FTS5 query with MATCH syntax
-2. Apply schema/table filters if specified
-3. Execute search with ranking (bm25)
-4. Return results with snippets highlighting matches
-5. Support pagination
-
-### 3. fts_list_indexes
-
-List all FTS indexes with metadata.
-
-**Parameters**: None
-
-**Response**:
-```json
-{
- "success": true,
- "indexes": [
- {
- "schema": "sales",
- "table": "orders",
- "columns": ["order_id", "customer_name", "notes"],
- "primary_key": "order_id",
- "row_count": 15000,
- "indexed_at": 1736668800
+ "kind": "table",
+ "key": "sales.orders",
+ "schema_name": "sales",
+ "object_name": "orders",
+ "content": "orders table with columns: order_id, customer_id, order_date, total_amount",
+ "rank": 0.5
}
]
}
```
**Implementation Logic**:
-1. Query `fts_indexes` table
-2. Return all indexes with metadata
+1. Search both `fts_objects` and `fts_llm` tables using FTS5
+2. Combine results with ranking
+3. Optionally fetch detailed object information
+4. Return ranked results
-### 4. fts_delete_index
+### 2. llm.search
-Remove an FTS index.
+Search LLM-generated content and insights using FTS5.
**Parameters**:
| Name | Type | Required | Description |
|------|------|----------|-------------|
-| schema | string | Yes | Schema name |
-| table | string | Yes | Table name |
+| query | string | Yes | FTS5 search query |
+| type | string | No | Content type to search ("summary", "relationship", "domain", "metric", "note") |
+| schema | string | No | Filter by schema |
+| limit | integer | No | Maximum results (default: 10) |
**Response**:
```json
{
"success": true,
- "schema": "sales",
- "table": "orders",
- "message": "Index deleted successfully"
+ "query": "customer segmentation",
+ "results": [
+ {
+ "kind": "domain",
+ "key": "customer_segmentation",
+ "content": "Customer segmentation based on purchase behavior and demographics",
+ "rank": 0.8
+ }
+ ]
}
```
**Implementation Logic**:
-1. Validate index exists
-2. Drop FTS search table
-3. Drop data table
-4. Remove metadata from `fts_indexes`
+1. Search `fts_llm` table using FTS5
+2. Apply filters if specified
+3. Return ranked results with content
-### 5. fts_reindex
+### 3. catalog_search (Detailed)
-Refresh an index with fresh data (full rebuild).
+Search indexed data using FTS5 across both database objects and LLM artifacts with detailed object information.
**Parameters**:
| Name | Type | Required | Description |
|------|------|----------|-------------|
-| schema | string | Yes | Schema name |
-| table | string | Yes | Table name |
-
-**Response**: Same as `fts_index_table`
-
-**Implementation Logic**:
-1. Fetch existing index metadata from `fts_indexes`
-2. Delete existing data from tables
-3. Call `index_table()` logic with stored metadata
-4. Update `indexed_at` timestamp
-
-### 6. fts_rebuild_all
-
-Rebuild ALL FTS indexes with fresh data.
-
-**Parameters**: None
+| query | string | Yes | FTS5 search query |
+| include_objects | boolean | No | Include detailed object information (default: false) |
+| object_limit | integer | No | Max objects to return when include_objects=true (default: 50) |
**Response**:
```json
{
"success": true,
- "rebuilt_count": 5,
- "failed": [],
- "indexes": [
+ "query": "customer order",
+ "results": [
{
- "schema": "sales",
- "table": "orders",
- "row_count": 15200,
- "status": "success"
+ "kind": "table",
+ "key": "sales.orders",
+ "schema_name": "sales",
+ "object_name": "orders",
+ "content": "orders table with columns: order_id, customer_id, order_date, total_amount",
+ "rank": 0.5,
+ "details": {
+ "object_id": 123,
+ "object_type": "table",
+ "schema_name": "sales",
+ "object_name": "orders",
+ "row_count_estimate": 15000,
+ "has_primary_key": true,
+ "has_foreign_keys": true,
+ "has_time_column": true,
+ "columns": [
+ {
+ "column_name": "order_id",
+ "data_type": "int",
+ "is_nullable": false,
+ "is_primary_key": true
+ }
+ ]
+ }
}
]
}
```
**Implementation Logic**:
-1. Get all indexes from `fts_indexes` table
-2. For each index:
- - Call `reindex()` with stored metadata
- - Track success/failure
-3. Return summary with rebuilt count and any failures
+1. Search both `fts_objects` and `fts_llm` tables using FTS5
+2. Combine results with ranking
+3. Optionally fetch detailed object information from `objects`, `columns`, `indexes`, `foreign_keys` tables
+4. Return ranked results with detailed information when requested
## Database Schema
-### fts_indexes (metadata table)
+### fts_objects (contentless FTS5 table)
```sql
-CREATE TABLE IF NOT EXISTS fts_indexes (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- schema_name TEXT NOT NULL,
- table_name TEXT NOT NULL,
- columns TEXT NOT NULL, -- JSON array of column names
- primary_key TEXT NOT NULL,
- where_clause TEXT,
- row_count INTEGER DEFAULT 0,
- indexed_at INTEGER DEFAULT (strftime('%s', 'now')),
- UNIQUE(schema_name, table_name)
+CREATE VIRTUAL TABLE fts_objects USING fts5(
+ schema_name,
+ object_name,
+ object_type,
+ content,
+ content='',
+ content_rowid='object_id'
);
-
-CREATE INDEX IF NOT EXISTS idx_fts_indexes_schema ON fts_indexes(schema_name);
-CREATE INDEX IF NOT EXISTS idx_fts_indexes_table ON fts_indexes(table_name);
```
-### Per-Index Tables (created dynamically)
-
-For each indexed table, create:
+### fts_llm (FTS5 table with content)
```sql
--- Data table (stores actual content)
-CREATE TABLE fts_data__ (
- rowid INTEGER PRIMARY KEY,
- content TEXT NOT NULL, -- Concatenated searchable text
- metadata TEXT -- JSON with original row data
-);
-
--- FTS5 virtual table (external content)
-CREATE VIRTUAL TABLE fts_search__ USING fts5(
- content,
- metadata,
- content='fts_data__',
- content_rowid='rowid',
- tokenize='porter unicode61'
+CREATE VIRTUAL TABLE fts_llm USING fts5(
+ kind,
+ key,
+ content
);
-
--- Triggers for automatic sync
-CREATE TRIGGER fts_ai_ AFTER INSERT ON fts_data_ BEGIN
- INSERT INTO fts_search_(rowid, content, metadata)
- VALUES (new.rowid, new.content, new.metadata);
-END;
-
-CREATE TRIGGER fts_ad_ AFTER DELETE ON fts_data_ BEGIN
- INSERT INTO fts_search_(fts_search_, rowid, content, metadata)
- VALUES ('delete', old.rowid, old.content, old.metadata);
-END;
-
-CREATE TRIGGER fts_au_ AFTER UPDATE ON fts_data_ BEGIN
- INSERT INTO fts_search_(fts_search_, rowid, content, metadata)
- VALUES ('delete', old.rowid, old.content, old.metadata);
- INSERT INTO fts_search_(rowid, content, metadata)
- VALUES (new.rowid, new.content, new.metadata);
-END;
```
-## Implementation Steps
-
-### Phase 1: Foundation
+## Implementation Status
-**Step 1: Create MySQL_FTS class**
-- Create `include/MySQL_FTS.h` - Class header with method declarations
-- Create `lib/MySQL_FTS.cpp` - Implementation
-- Follow `MySQL_Catalog` pattern for SQLite management
+### Phase 1: Foundation ✅ COMPLETED
-**Step 2: Add configuration variable**
-- Modify `include/MCP_Thread.h` - Add `mcp_fts_path` to variables struct
-- Modify `lib/MCP_Thread.cpp` - Add to `mcp_thread_variables_names` array
-- Handle `fts_path` in get/set variable functions
-- Default value: `"mcp_fts.db"`
+**Step 1: Integrate FTS into Discovery_Schema**
+- FTS functionality built into `lib/Discovery_Schema.cpp`
+- Uses existing `mcp_catalog.db` database
+- No separate configuration variable needed
-**Step 3: Integrate FTS into MySQL_Tool_Handler**
-- Add `MySQL_FTS* fts` member to `include/MySQL_Tool_Handler.h`
-- Initialize in constructor with `fts_path`
-- Clean up in destructor
-- Add FTS tool method declarations
+**Step 2: Create FTS tables**
+- `fts_objects` for database objects (contentless)
+- `fts_llm` for LLM artifacts (with content)
-### Phase 2: Core Indexing
+### Phase 2: Core Indexing ✅ COMPLETED
-**Step 4: Implement fts_index_table tool**
-```cpp
-// In MySQL_FTS class
-std::string index_table(
- const std::string& schema,
- const std::string& table,
- const std::string& columns, // JSON array
- const std::string& primary_key,
- const std::string& where_clause,
- MySQL_Tool_Handler* mysql_handler
-);
-```
+**Step 3: Implement automatic indexing**
+- Objects automatically indexed during static harvest
+- LLM artifacts automatically indexed during upsert operations
-Logic:
-- Parse columns JSON array
-- Create sanitized table name (replace dots/underscores)
-- Create `fts_data_*` and `fts_search_*` tables
-- Fetch data: `mysql_handler->execute_query(sql)`
-- Build content by concatenating column values
-- Insert in batches for performance
-- Update metadata
+### Phase 3: Search Functionality ✅ COMPLETED
-**Step 5: Implement fts_list_indexes tool**
-```cpp
-std::string list_indexes();
-```
-Query `fts_indexes` and return JSON array.
+**Step 4: Implement search tools**
+- `catalog_search` tool in Query_Tool_Handler
+- `llm.search` tool in Query_Tool_Handler
-**Step 6: Implement fts_delete_index tool**
-```cpp
-std::string delete_index(const std::string& schema, const std::string& table);
-```
-Drop tables and remove metadata.
-
-### Phase 3: Search Functionality
-
-**Step 7: Implement fts_search tool**
-```cpp
-std::string search(
- const std::string& query,
- const std::string& schema,
- const std::string& table,
- int limit,
- int offset
-);
-```
-
-SQL query template:
-```sql
-SELECT
- d.schema_name,
- d.table_name,
- d.primary_key_value,
- snippet(fts_search, 2, '', '', '...', 30) as snippet,
- d.metadata
-FROM fts_search s
-JOIN fts_data d ON s.rowid = d.rowid
-WHERE fts_search MATCH ?
-ORDER BY bm25(fts_search)
-LIMIT ? OFFSET ?
-```
-
-**Step 8: Implement fts_reindex tool**
-```cpp
-std::string reindex(
- const std::string& schema,
- const std::string& table,
- MySQL_Tool_Handler* mysql_handler
-);
-```
-Fetch metadata, delete old data, rebuild.
+### Phase 4: Tool Registration ✅ COMPLETED
-**Step 9: Implement fts_rebuild_all tool**
-```cpp
-std::string rebuild_all(MySQL_Tool_Handler* mysql_handler);
-```
-Loop through all indexes and rebuild each.
-
-### Phase 4: Tool Registration
-
-**Step 10: Register tools in Query_Tool_Handler**
-- Modify `lib/Query_Tool_Handler.cpp`
-- Add to `get_tool_list()`:
- ```cpp
- tools.push_back(create_tool_schema(
- "fts_index_table",
- "Create/populate FTS index for a table",
- {"schema", "table", "columns", "primary_key"},
- {{"where_clause", "string"}}
- ));
- // Repeat for all 6 tools
- ```
-- Add routing in `execute_tool()`:
- ```cpp
- else if (tool_name == "fts_index_table") {
- std::string schema = get_json_string(arguments, "schema");
- std::string table = get_json_string(arguments, "table");
- std::string columns = get_json_string(arguments, "columns");
- std::string primary_key = get_json_string(arguments, "primary_key");
- std::string where_clause = get_json_string(arguments, "where_clause");
- result_str = mysql_handler->fts_index_table(schema, table, columns, primary_key, where_clause);
- }
- // Repeat for other tools
- ```
-
-**Step 11: Update ProxySQL_MCP_Server**
-- Modify `lib/ProxySQL_MCP_Server.cpp`
-- Pass `fts_path` when creating MySQL_Tool_Handler
-- Initialize FTS: `mysql_handler->get_fts()->init()`
-
-### Phase 5: Build and Test
-
-**Step 12: Update build system**
-- Modify `Makefile`
-- Add `lib/MySQL_FTS.cpp` to compilation sources
-- Verify link against sqlite3
-
-**Step 13: Testing**
-- Test all 6 tools via MCP endpoint
-- Verify JSON responses
-- Test with actual MySQL data
-- Test cross-table search
-- Test WHERE clause filtering
+**Step 5: Register tools**
+- Tools registered in Query_Tool_Handler::get_tool_list()
+- Tools routed in Query_Tool_Handler::execute_tool()
## Critical Files
-### New Files to Create
-- `include/MySQL_FTS.h` - FTS class header
-- `lib/MySQL_FTS.cpp` - FTS class implementation
-
-### Files to Modify
-- `include/MySQL_Tool_Handler.h` - Add FTS member and tool method declarations
-- `lib/MySQL_Tool_Handler.cpp` - Add FTS tool wrappers, initialize FTS
-- `lib/Query_Tool_Handler.cpp` - Register and route FTS tools
-- `include/MCP_Thread.h` - Add `mcp_fts_path` variable
-- `lib/MCP_Thread.cpp` - Handle `fts_path` configuration
-- `lib/ProxySQL_MCP_Server.cpp` - Pass `fts_path` to MySQL_Tool_Handler
-- `Makefile` - Add MySQL_FTS.cpp to build
+### Files Modified
+- `include/Discovery_Schema.h` - Added FTS methods
+- `lib/Discovery_Schema.cpp` - Implemented FTS functionality
+- `lib/Query_Tool_Handler.cpp` - Added FTS tool routing
+- `include/Query_Tool_Handler.h` - Added FTS tool declarations
-## Code Patterns to Follow
+## Current Implementation Details
-### MySQL_FTS Class Structure (similar to MySQL_Catalog)
+### FTS Integration Pattern
```cpp
-class MySQL_FTS {
+class Discovery_Schema {
private:
- SQLite3DB* db;
- std::string db_path;
-
- int init_schema();
- int create_tables();
- int create_index_tables(const std::string& schema, const std::string& table);
- std::string get_data_table_name(const std::string& schema, const std::string& table);
- std::string get_fts_table_name(const std::string& schema, const std::string& table);
-
+ // FTS methods
+ int create_fts_tables();
+ int rebuild_fts_index(int run_id);
+ json search_fts(const std::string& query, bool include_objects = false, int object_limit = 50);
+ json search_llm_fts(const std::string& query, const std::string& type = "",
+ const std::string& schema = "", int limit = 10);
+
public:
- MySQL_FTS(const std::string& path);
- ~MySQL_FTS();
-
- int init();
- void close();
-
- // Tool methods
- std::string index_table(...);
- std::string search(...);
- std::string list_indexes();
- std::string delete_index(...);
- std::string reindex(...);
- std::string rebuild_all(...);
-
- bool index_exists(const std::string& schema, const std::string& table);
- SQLite3DB* get_db() { return db; }
+ // FTS is automatically maintained during:
+ // - Object insertion (static harvest)
+ // - LLM artifact upsertion
+ // - Catalog rebuild operations
};
```
@@ -477,22 +250,22 @@ public:
json result;
result["success"] = false;
result["error"] = "Descriptive error message";
-return result.dump();
+return result;
// Logging
proxy_error("FTS error: %s\n", error_msg);
-proxy_info("FTS index created: %s.%s\n", schema.c_str(), table.c_str());
+proxy_info("FTS search completed: %zu results\n", result_count);
```
### SQLite Operations Pattern
```cpp
db->wrlock();
-// Write operations
+// Write operations (indexing)
db->wrunlock();
db->rdlock();
-// Read operations
+// Read operations (search)
db->rdunlock();
// Prepared statements
@@ -503,80 +276,60 @@ SAFE_SQLITE3_STEP2(stmt);
(*proxy_sqlite3_finalize)(stmt);
```
-### JSON Response Pattern
-
-```cpp
-// Use nlohmann/json
-json result;
-result["success"] = true;
-result["data"] = data_array;
-return result.dump();
-```
-
-## Configuration Variable
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `mcp-ftspath` | `mcp_fts.db` | Path to FTS SQLite database (relative or absolute) |
-
-**Usage**:
-```sql
-SET mcp-ftspath='/var/lib/proxysql/mcp_fts.db';
-```
-
## Agent Workflow Example
```python
-# Agent narrows down results using FTS
-fts_results = call_tool("fts_search", {
- "query": "urgent customer complaint",
- "limit": 10
+# Agent searches for relevant objects
+search_results = call_tool("catalog_search", {
+ "query": "customer orders with high value",
+ "include_objects": True,
+ "object_limit": 20
})
-# Extract primary keys from FTS results
-order_ids = [r["primary_key_value"] for r in fts_results["results"]]
-
-# Query MySQL for full data
-full_data = call_tool("run_sql_readonly", {
- "sql": f"SELECT * FROM orders WHERE order_id IN ({','.join(order_ids)})"
+# Agent searches for LLM insights
+llm_results = call_tool("llm.search", {
+ "query": "customer segmentation",
+ "type": "domain"
})
+
+# Agent uses results to build understanding
+for result in search_results["results"]:
+ if result["kind"] == "table":
+ # Get detailed table information
+ table_details = call_tool("catalog_get_object", {
+ "schema": result["schema_name"],
+ "object": result["object_name"]
+ })
```
-## Threading Considerations
+## Performance Considerations
-- SQLite3DB provides thread-safe read-write locks
-- Use `wrlock()` for writes (index operations)
-- Use `rdlock()` for reads (search operations)
-- Follow the catalog pattern for locking
+1. **Contentless FTS**: `fts_objects` uses contentless indexing for performance
+2. **Automatic Maintenance**: FTS indexes automatically maintained during operations
+3. **Ranking**: Results ranked using FTS5 bm25 algorithm
+4. **Pagination**: Large result sets automatically paginated
-## Performance Considerations
+## Testing Status ✅ COMPLETED
-1. **Batch inserts**: When indexing, insert rows in batches (100-1000 at a time)
-2. **Table naming**: Sanitize schema/table names for SQLite table names
-3. **Memory usage**: Large tables may require streaming results
-4. **Index size**: Monitor FTS database size
-
-## Testing Checklist
-
-- [ ] Create index on single table
-- [ ] Create index with WHERE clause
-- [ ] Search single table
-- [ ] Search across all tables
-- [ ] List indexes
-- [ ] Delete index
-- [ ] Reindex single table
-- [ ] Rebuild all indexes
-- [ ] Test with NULL values
-- [ ] Test with special characters in data
-- [ ] Test pagination
-- [ ] Test schema/table filtering
+- [x] Search database objects using FTS
+- [x] Search LLM artifacts using FTS
+- [x] Combined search with ranking
+- [x] Detailed object information retrieval
+- [x] Filter by content type
+- [x] Filter by schema
+- [x] Performance with large catalogs
+- [x] Error handling
## Notes
-- Follow existing patterns from `MySQL_Catalog` for SQLite management
-- Use SQLite3DB read-write locks for thread safety
-- Return JSON responses using nlohmann/json library
-- Handle NULL values properly (use empty string as in execute_query)
-- Use prepared statements for SQL safety
-- Log errors using `proxy_error()` and info using `proxy_info()`
-- Table name sanitization: replace `.` and special chars with `_`
+- FTS5 requires SQLite with FTS5 extension enabled
+- Contentless FTS for objects provides fast search without duplicating data
+- LLM artifacts stored directly in FTS table for full content search
+- Automatic FTS maintenance ensures indexes are always current
+- Ranking uses FTS5's built-in bm25 algorithm for relevance scoring
+
+## Version
+
+- **Last Updated:** 2026-01-19
+- **Implementation Date:** January 2026
+- **Status:** Fully implemented and tested
diff --git a/doc/MCP/Tool_Discovery_Guide.md b/doc/MCP/Tool_Discovery_Guide.md
index aaa2f38ff3..113af68f48 100644
--- a/doc/MCP/Tool_Discovery_Guide.md
+++ b/doc/MCP/Tool_Discovery_Guide.md
@@ -1,6 +1,6 @@
# MCP Tool Discovery Guide
-This guide explains how to discover and interact with MCP tools available on the Query endpoint.
+This guide explains how to discover and interact with MCP tools available on all endpoints, with a focus on the Query endpoint which includes database exploration and two-phase discovery tools.
## Overview
@@ -258,6 +258,143 @@ Delete an entry from the catalog.
- `kind` (string, **required**) - Entry kind
- `key` (string, **required**) - Entry key
+### Two-Phase Discovery Tools
+
+#### discovery.run_static
+Run Phase 1 of two-phase discovery: static harvest of database metadata.
+
+**Parameters:**
+- `schema_filter` (string, optional) - Filter schemas by name pattern
+- `table_filter` (string, optional) - Filter tables by name pattern
+- `run_id` (string, optional) - Custom run identifier
+
+**Returns:**
+- `run_id` - Unique identifier for this discovery run
+- `objects_count` - Number of database objects discovered
+- `schemas_count` - Number of schemas processed
+- `tables_count` - Number of tables processed
+- `columns_count` - Number of columns processed
+- `indexes_count` - Number of indexes processed
+- `constraints_count` - Number of constraints processed
+
+#### agent.run_start
+Start a new agent run for discovery coordination.
+
+**Parameters:**
+- `run_id` (string, **required**) - Discovery run identifier
+- `agent_id` (string, **required**) - Agent identifier
+- `capabilities` (array, optional) - List of agent capabilities
+
+#### agent.run_finish
+Mark an agent run as completed.
+
+**Parameters:**
+- `run_id` (string, **required**) - Discovery run identifier
+- `agent_id` (string, **required**) - Agent identifier
+- `status` (string, **required**) - Final status ("success", "error", "timeout")
+- `summary` (string, optional) - Summary of work performed
+
+#### agent.event_append
+Append an event to an agent run.
+
+**Parameters:**
+- `run_id` (string, **required**) - Discovery run identifier
+- `agent_id` (string, **required**) - Agent identifier
+- `event_type` (string, **required**) - Type of event
+- `data` (object, **required**) - Event data
+- `timestamp` (string, optional) - ISO8601 timestamp
+
+### LLM Interaction Tools
+
+#### llm.summary_upsert
+Store or update a table/column summary generated by LLM.
+
+**Parameters:**
+- `schema` (string, **required**) - Schema name
+- `table` (string, **required**) - Table name
+- `column` (string, optional) - Column name (if column-level summary)
+- `summary` (string, **required**) - LLM-generated summary
+- `confidence` (number, optional) - Confidence score (0.0-1.0)
+
+#### llm.summary_get
+Retrieve LLM-generated summary for a table or column.
+
+**Parameters:**
+- `schema` (string, **required**) - Schema name
+- `table` (string, **required**) - Table name
+- `column` (string, optional) - Column name
+
+#### llm.relationship_upsert
+Store or update an inferred relationship between tables.
+
+**Parameters:**
+- `source_schema` (string, **required**) - Source schema
+- `source_table` (string, **required**) - Source table
+- `target_schema` (string, **required**) - Target schema
+- `target_table` (string, **required**) - Target table
+- `confidence` (number, **required**) - Confidence score (0.0-1.0)
+- `description` (string, **required**) - Relationship description
+- `type` (string, optional) - Relationship type ("fk", "semantic", "usage")
+
+#### llm.domain_upsert
+Store or update a business domain classification.
+
+**Parameters:**
+- `domain_id` (string, **required**) - Domain identifier
+- `name` (string, **required**) - Domain name
+- `description` (string, **required**) - Domain description
+- `confidence` (number, optional) - Confidence score (0.0-1.0)
+- `tags` (array, optional) - Domain tags
+
+#### llm.domain_set_members
+Set the members (tables) of a business domain.
+
+**Parameters:**
+- `domain_id` (string, **required**) - Domain identifier
+- `members` (array, **required**) - List of table identifiers
+- `confidence` (number, optional) - Confidence score (0.0-1.0)
+
+#### llm.metric_upsert
+Store or update a business metric definition.
+
+**Parameters:**
+- `metric_id` (string, **required**) - Metric identifier
+- `name` (string, **required**) - Metric name
+- `description` (string, **required**) - Metric description
+- `formula` (string, **required**) - SQL formula or description
+- `domain_id` (string, optional) - Associated domain
+- `tags` (array, optional) - Metric tags
+
+#### llm.question_template_add
+Add a question template that can be answered using this data.
+
+**Parameters:**
+- `template_id` (string, **required**) - Template identifier
+- `question` (string, **required**) - Question template with placeholders
+- `answer_plan` (object, **required**) - Steps to answer the question
+- `complexity` (string, optional) - Complexity level ("low", "medium", "high")
+- `estimated_time` (number, optional) - Estimated time in minutes
+- `tags` (array, optional) - Template tags
+
+#### llm.note_add
+Add a general note or insight about the data.
+
+**Parameters:**
+- `note_id` (string, **required**) - Note identifier
+- `content` (string, **required**) - Note content
+- `type` (string, optional) - Note type ("insight", "warning", "recommendation")
+- `confidence` (number, optional) - Confidence score (0.0-1.0)
+- `tags` (array, optional) - Note tags
+
+#### llm.search
+Search LLM-generated content and insights.
+
+**Parameters:**
+- `query` (string, **required**) - Search query
+- `type` (string, optional) - Content type to search ("summary", "relationship", "domain", "metric", "note")
+- `schema` (string, optional) - Filter by schema
+- `limit` (number, optional) - Maximum results (default: 10)
+
## Calling a Tool
### Request Format
@@ -455,10 +592,11 @@ The test script provides a convenient way to discover and test tools:
The same discovery pattern works for all MCP endpoints:
- **Config**: `/mcp/config` - Configuration management tools
-- **Query**: `/mcp/query` - Database exploration and query tools
+- **Query**: `/mcp/query` - Database exploration, query, and discovery tools
- **Admin**: `/mcp/admin` - Administrative operations
- **Cache**: `/mcp/cache` - Cache management tools
- **Observe**: `/mcp/observe` - Monitoring and metrics tools
+- **AI**: `/mcp/ai` - AI and LLM features
Simply change the endpoint URL:
@@ -470,6 +608,10 @@ curl -k -X POST https://127.0.0.1:6071/mcp/config \
## Related Documentation
-- [Architecture.md](Architecture.md) - Overall MCP architecture
-- [Database_Discovery_Agent.md](Database_Discovery_Agent.md) - AI agent architecture
-- [README.md](README.md) - Module overview
+- [Architecture.md](Architecture.md) - Overall MCP architecture and endpoint specifications
+- [VARIABLES.md](VARIABLES.md) - Configuration variables reference
+
+## Version
+
+- **Last Updated:** 2026-01-19
+- **MCP Protocol:** JSON-RPC 2.0 over HTTPS
diff --git a/doc/MCP/VARIABLES.md b/doc/MCP/VARIABLES.md
index 92edc552e6..ceede8c046 100644
--- a/doc/MCP/VARIABLES.md
+++ b/doc/MCP/VARIABLES.md
@@ -4,7 +4,7 @@ This document describes all configuration variables for the MCP (Model Context P
## Overview
-The MCP module provides JSON-RPC 2.0 over HTTPS for LLM integration with ProxySQL. It includes endpoints for configuration, observation, querying, administration, caching, and a MySQL Tool Handler for database exploration.
+The MCP module provides JSON-RPC 2.0 over HTTPS for LLM integration with ProxySQL. It includes endpoints for configuration, observation, querying, administration, caching, and AI features, each with dedicated tool handlers for database exploration and LLM integration.
All variables are stored in the `global_variables` table with the `mcp-` prefix and can be modified at runtime through the admin interface.
@@ -106,9 +106,20 @@ The following variables control authentication (Bearer tokens) for specific MCP
LOAD MCP VARIABLES TO RUNTIME;
```
-### MySQL Tool Handler Configuration
+#### `mcp-ai_endpoint_auth`
+- **Type:** String
+- **Default:** `""` (empty)
+- **Description:** Bearer token for `/mcp/ai` endpoint
+- **Runtime:** Yes
+- **Example:**
+ ```sql
+ SET mcp-ai_endpoint_auth='ai-token';
+ LOAD MCP VARIABLES TO RUNTIME;
+ ```
-The MySQL Tool Handler provides LLM-based tools for MySQL database exploration, including:
+### Query Tool Handler Configuration
+
+The Query Tool Handler provides LLM-based tools for MySQL database exploration and two-phase discovery, including:
- **inventory** - List databases and tables
- **structure** - Get table schema
- **profiling** - Analyze query performance
@@ -116,6 +127,9 @@ The MySQL Tool Handler provides LLM-based tools for MySQL database exploration,
- **query** - Execute SQL queries
- **relationships** - Infer table relationships
- **catalog** - Catalog operations
+- **discovery** - Two-phase discovery tools (static harvest + LLM analysis)
+- **agent** - Agent coordination tools
+- **llm** - LLM interaction tools
#### `mcp-mysql_hosts`
- **Type:** String (comma-separated)
@@ -175,16 +189,11 @@ The MySQL Tool Handler provides LLM-based tools for MySQL database exploration,
### Catalog Configuration
-#### `mcp-catalog_path`
-- **Type:** String (file path)
-- **Default:** `"mcp_catalog.db"`
-- **Description:** Path to the SQLite catalog database (relative to ProxySQL datadir)
-- **Runtime:** Yes
-- **Example:**
- ```sql
- SET mcp-catalog_path='/path/to/mcp_catalog.db';
- LOAD MCP VARIABLES TO RUNTIME;
- ```
+The catalog database path is **hardcoded** to `mcp_catalog.db` in the ProxySQL datadir and cannot be changed at runtime. The catalog stores:
+- Database schemas discovered during two-phase discovery
+- LLM memories (summaries, domains, metrics)
+- Tool usage statistics
+- Search history
## Management Commands
@@ -271,9 +280,9 @@ SELECT * FROM stats_mysql_global WHERE variable_name LIKE 'mcp_%';
- **MCP Thread Version:** 0.1.0
- **Protocol:** JSON-RPC 2.0 over HTTPS
+- **Last Updated:** 2026-01-19
## Related Documentation
-- [MCP Module README](README.md) - Module overview and setup
-- [MCP Endpoints](ENDPOINTS.md) - API endpoint documentation
-- [MySQL Tool Handler](TOOL_HANDLER.md) - Tool-specific documentation
+- [MCP Architecture](Architecture.md) - Module architecture and endpoint specifications
+- [Tool Discovery Guide](Tool_Discovery_Guide.md) - Tool discovery and usage documentation
diff --git a/doc/MCP/Vector_Embeddings_Implementation_Plan.md b/doc/MCP/Vector_Embeddings_Implementation_Plan.md
index 0be878068a..a9853f4fea 100644
--- a/doc/MCP/Vector_Embeddings_Implementation_Plan.md
+++ b/doc/MCP/Vector_Embeddings_Implementation_Plan.md
@@ -1,8 +1,10 @@
-# Vector Embeddings Implementation Plan
+# Vector Embeddings Implementation Plan (NOT YET IMPLEMENTED)
## Overview
-This document describes the implementation of Vector Embeddings capabilities for the ProxySQL MCP Query endpoint. The Embeddings system enables AI agents to perform semantic similarity searches on database content using sqlite-vec for vector storage and sqlite-rembed for embedding generation.
+This document describes the planned implementation of Vector Embeddings capabilities for the ProxySQL MCP Query endpoint. The Embeddings system will enable AI agents to perform semantic similarity searches on database content using sqlite-vec for vector storage and sqlite-rembed for embedding generation.
+
+**Status: PLANNED** ⏳
## Requirements
@@ -19,21 +21,19 @@ MCP Query Endpoint (JSON-RPC 2.0 over HTTPS)
↓
Query_Tool_Handler (routes tool calls)
↓
-MySQL_Tool_Handler (implements tools)
- ↓
-MySQL_Embeddings (new class - manages embeddings database)
+Discovery_Schema (manages embeddings database)
↓
-SQLite with sqlite-vec (mcp_embeddings.db)
+SQLite with sqlite-vec (mcp_catalog.db)
↓
-sqlite-rembed (embedding generation)
+LLM_Bridge (embedding generation)
↓
External APIs (OpenAI, Ollama, Cohere, etc.)
```
## Database Design
-### Separate SQLite Database
-**Path**: `mcp_embeddings.db` (configurable via `mcp-embeddingpath` variable)
+### Integrated with Discovery Schema
+**Path**: `mcp_catalog.db` (uses existing catalog database)
### Schema
@@ -147,738 +147,116 @@ SELECT
COALESCE(customer_name, '') || ' ' ||
COALESCE(product_name, '') || ' ' ||
COALESCE(notes, '')) as vector,
- CAST(order_id AS TEXT) as pk_value,
- json_object(
- 'order_id', order_id,
- 'customer_name', customer_name,
- 'notes', notes
- ) as metadata
-FROM testdb.orders
-WHERE active = 1;
-```
-
-### 2. embed_search
-
-Perform semantic similarity search using vector embeddings.
-
-**Parameters**:
-| Name | Type | Required | Description |
-|------|------|----------|-------------|
-| query | string | Yes | Search query text |
-| schema | string | No | Filter by schema |
-| table | string | No | Filter by table |
-| limit | integer | No | Max results (default: 10) |
-| min_distance | float | No | Maximum distance threshold (default: 1.0) |
-
-**Response**:
-```json
-{
- "success": true,
- "query": "customer complaining about late delivery",
- "query_embedding_dim": 1536,
- "total_matches": 25,
- "results": [
- {
- "schema": "testdb",
- "table": "orders",
- "primary_key_value": "12345",
- "distance": 0.234,
- "metadata": {
- "order_id": 12345,
- "customer_name": "John Doe",
- "notes": "Customer upset about delivery delay"
- }
- }
- ]
-}
-```
-
-**Implementation Logic**:
-1. Generate embedding for query text using `rembed()`
-2. Build SQL with vector similarity search
-3. Apply schema/table filters if specified
-4. Execute KNN search with distance threshold
-5. Return ranked results with metadata
-
-**SQL Query Template**:
-```sql
-SELECT
- e.pk_value as primary_key_value,
- e.distance,
- e.metadata
-FROM embeddings_testdb_orders e
-WHERE e.vector MATCH rembed('mcp_embeddings', ?)
- AND e.distance < ?
-ORDER BY e.distance ASC
-LIMIT ?;
-```
-**Distance Metrics** (sqlite-vec supports):
-- L2 (Euclidean) - default
-- Cosine - for normalized vectors
-- Hamming - for binary vectors
+## Implementation Status
-### 3. embed_list_indexes
+### Phase 1: Foundation ⏳ PLANNED
-List all embedding indexes with metadata.
+**Step 1: Integrate Embeddings into Discovery_Schema**
+- Embeddings functionality to be built into `lib/Discovery_Schema.cpp`
+- Will use existing `mcp_catalog.db` database
+- Will require new configuration variable `mcp-embeddingpath`
-**Parameters**: None
+**Step 2: Create Embeddings tables**
+- `embedding_indexes` for metadata
+- `embedding_data__` for vector storage
+- Integration with sqlite-vec extension
-**Response**:
-```json
-{
- "success": true,
- "indexes": [
- {
- "schema": "testdb",
- "table": "orders",
- "columns": ["customer_name", "product_name", "notes"],
- "primary_key": "order_id",
- "model": "text-embedding-3-small",
- "vector_dim": 1536,
- "strategy": "concat",
- "row_count": 5000,
- "indexed_at": 1736668800
- }
- ]
-}
-```
+### Phase 2: Core Indexing ⏳ PLANNED
-**Implementation Logic**:
-1. Query `embedding_indexes` table
-2. Return all indexes with metadata
+**Step 3: Implement embedding generation**
+- Integration with LLM_Bridge for embedding generation
+- Support for multiple embedding models
+- Batch processing for performance
-### 4. embed_delete_index
+### Phase 3: Search Functionality ⏳ PLANNED
-Remove an embedding index.
+**Step 4: Implement search tools**
+- `embedding_search` tool in Query_Tool_Handler
+- Semantic similarity search with ranking
-**Parameters**:
-| Name | Type | Required | Description |
-|------|------|----------|-------------|
-| schema | string | Yes | Schema name |
-| table | string | Yes | Table name |
+### Phase 4: Tool Registration ⏳ PLANNED
-**Response**:
-```json
-{
- "success": true,
- "schema": "testdb",
- "table": "orders",
- "message": "Embedding index deleted successfully"
-}
-```
+**Step 5: Register tools**
+- Tools to be registered in Query_Tool_Handler::get_tool_list()
+- Tools to be routed in Query_Tool_Handler::execute_tool()
-**Implementation Logic**:
-1. Validate index exists
-2. Drop vec0 table
-3. Remove metadata from `embedding_indexes`
-
-### 5. embed_reindex
-
-Refresh an embedding index with fresh data (full rebuild).
-
-**Parameters**:
-| Name | Type | Required | Description |
-|------|------|----------|-------------|
-| schema | string | Yes | Schema name |
-| table | string | Yes | Table name |
-
-**Response**: Same as `embed_index_table`
-
-**Implementation Logic**:
-1. Fetch existing index metadata from `embedding_indexes`
-2. Drop existing vec0 table
-3. Re-create vec0 table
-4. Call `embed_index_table` logic with stored metadata
-5. Update `indexed_at` timestamp
-
-### 6. embed_rebuild_all
-
-Rebuild ALL embedding indexes with fresh data.
-
-**Parameters**: None
-
-**Response**:
-```json
-{
- "success": true,
- "rebuilt_count": 3,
- "failed": [
- {
- "schema": "testdb",
- "table": "products",
- "error": "API rate limit exceeded"
- }
- ],
- "indexes": [
- {
- "schema": "testdb",
- "table": "orders",
- "row_count": 5100,
- "status": "success"
- }
- ]
-}
-```
-
-**Implementation Logic**:
-1. Get all indexes from `embedding_indexes` table
-2. For each index:
- - Call `reindex()` with stored metadata
- - Track success/failure
-3. Return summary with rebuilt count and any failures
-
-## Implementation Steps
-
-### Phase 1: Foundation
-
-**Step 1: Create MySQL_Embeddings class**
-- Create `include/MySQL_Embeddings.h` - Class header with method declarations
-- Create `lib/MySQL_Embeddings.cpp` - Implementation
-- Follow `MySQL_FTS` and `MySQL_Catalog` patterns
-
-**Step 2: Add configuration variable**
-- Modify `include/MCP_Thread.h` - Add `mcp_embedding_path` to variables struct
-- Modify `lib/MCP_Thread.cpp` - Add to `mcp_thread_variables_names` array
-- Handle `embedding_path` in get/set variable functions
-- Default value: `"mcp_embeddings.db"`
-
-**Step 3: Integrate Embeddings into MySQL_Tool_Handler**
-- Add `MySQL_Embeddings* embeddings` member to `include/MySQL_Tool_Handler.h`
-- Initialize in constructor with `embedding_path`
-- Clean up in destructor
-- Add Embeddings tool method declarations
-
-### Phase 2: Core Indexing
-
-**Step 4: Implement embed_index_table tool**
-```cpp
-// In MySQL_Embeddings class
-std::string index_table(
- const std::string& schema,
- const std::string& table,
- const std::string& columns, // JSON array
- const std::string& primary_key,
- const std::string& where_clause,
- const std::string& model,
- const std::string& strategy,
- MySQL_Tool_Handler* mysql_handler
-);
-```
-
-Key implementation details:
-- Parse columns JSON array
-- Create sanitized table name
-- Create vec0 table with appropriate dimensions
-- Configure sqlite-rembed client if needed
-- Fetch data from MySQL
-- Generate embeddings using `rembed()` function
-- Insert into vec0 table
-- Update metadata
-
-**GenAI Module Placeholder**:
-```cpp
-// For future GenAI module integration
-// Currently uses sqlite-rembed
-std::vector generate_embedding(
- const std::string& text,
- const std::string& model
-) {
- // PLACEHOLDER: Will call GenAI module when merged
- // Currently: Use sqlite-rembed
-
- char* error = NULL;
- std::string sql = "SELECT rembed('mcp_embeddings', ?) as embedding";
-
- // Execute query, parse JSON array
- // Return std::vector
-}
-```
-
-**Step 5: Implement embed_list_indexes tool**
-```cpp
-std::string list_indexes();
-```
-Query `embedding_indexes` and return JSON array.
+## Critical Files (PLANNED)
-**Step 6: Implement embed_delete_index tool**
-```cpp
-std::string delete_index(const std::string& schema, const std::string& table);
-```
-Drop vec0 table and remove metadata.
-
-### Phase 3: Search Functionality
-
-**Step 7: Implement embed_search tool**
-```cpp
-std::string search(
- const std::string& query,
- const std::string& schema,
- const std::string& table,
- int limit,
- float min_distance
-);
-```
-
-SQL query template:
-```sql
-SELECT
- e.pk_value,
- e.distance,
- e.metadata
-FROM embeddings_ e
-WHERE e.vector MATCH rembed('mcp_embeddings', ?)
- AND e.distance < ?
-ORDER BY e.distance ASC
-LIMIT ?;
-```
-
-**Step 8: Implement embed_reindex tool**
-```cpp
-std::string reindex(
- const std::string& schema,
- const std::string& table,
- MySQL_Tool_Handler* mysql_handler
-);
-```
-Fetch metadata, rebuild embeddings.
-
-**Step 9: Implement embed_rebuild_all tool**
-```cpp
-std::string rebuild_all(MySQL_Tool_Handler* mysql_handler);
-```
-Loop through all indexes and rebuild each.
-
-### Phase 4: Tool Registration
-
-**Step 10: Register tools in Query_Tool_Handler**
-- Modify `lib/Query_Tool_Handler.cpp`
-- Add to `get_tool_list()`:
- ```cpp
- tools.push_back(create_tool_schema(
- "embed_index_table",
- "Generate embeddings and create vector index for a table",
- {"schema", "table", "columns", "primary_key", "model"},
- {{"where_clause", "string"}, {"strategy", "string"}}
- ));
- // Repeat for all 6 tools
- ```
-- Add routing in `execute_tool()`:
- ```cpp
- else if (tool_name == "embed_index_table") {
- std::string schema = get_json_string(arguments, "schema");
- std::string table = get_json_string(arguments, "table");
- std::string columns = get_json_string(arguments, "columns");
- std::string primary_key = get_json_string(arguments, "primary_key");
- std::string where_clause = get_json_string(arguments, "where_clause");
- std::string model = get_json_string(arguments, "model");
- std::string strategy = get_json_string(arguments, "strategy", "concat");
- result_str = mysql_handler->embed_index_table(schema, table, columns, primary_key, where_clause, model, strategy);
- }
- // Repeat for other tools
- ```
-
-**Step 11: Update ProxySQL_MCP_Server**
-- Modify `lib/ProxySQL_MCP_Server.cpp`
-- Pass `embedding_path` when creating MySQL_Tool_Handler
-- Initialize Embeddings: `mysql_handler->get_embeddings()->init()`
-
-### Phase 5: Build and Test
-
-**Step 12: Update build system**
-- Modify `Makefile`
-- Add `lib/MySQL_Embeddings.cpp` to compilation sources
-- Verify link against sqlite3 (already includes vec.o)
-
-**Step 13: Testing**
-- Test all 6 embed tools via MCP endpoint
-- Verify JSON responses
-- Test with actual MySQL data
-- Test cross-table semantic search
-- Test different embedding strategies
-- Test with sqlite-rembed configured
-
-## Critical Files
-
-### New Files to Create
+### Files to Create
- `include/MySQL_Embeddings.h` - Embeddings class header
- `lib/MySQL_Embeddings.cpp` - Embeddings class implementation
### Files to Modify
-- `include/MySQL_Tool_Handler.h` - Add embeddings member and tool method declarations
-- `lib/MySQL_Tool_Handler.cpp` - Add embeddings tool wrappers, initialize embeddings
-- `lib/Query_Tool_Handler.cpp` - Register and route embeddings tools
+- `include/Discovery_Schema.h` - Add Embeddings methods
+- `lib/Discovery_Schema.cpp` - Implement Embeddings functionality
+- `lib/Query_Tool_Handler.cpp` - Add Embeddings tool routing
+- `include/Query_Tool_Handler.h` - Add Embeddings tool declarations
- `include/MCP_Thread.h` - Add `mcp_embedding_path` variable
- `lib/MCP_Thread.cpp` - Handle `embedding_path` configuration
-- `lib/ProxySQL_MCP_Server.cpp` - Pass `embedding_path` to MySQL_Tool_Handler
+- `lib/ProxySQL_MCP_Server.cpp` - Pass `embedding_path` to components
- `Makefile` - Add MySQL_Embeddings.cpp to build
-## Code Patterns to Follow
+## Future Implementation Details
-### MySQL_Embeddings Class Structure
+### Embeddings Integration Pattern
```cpp
-class MySQL_Embeddings {
+class Discovery_Schema {
private:
- SQLite3DB* db;
- std::string db_path;
-
- // Schema management
- int init_schema();
- int create_tables();
- int create_embedding_table(const std::string& schema,
- const std::string& table,
- int vector_dim);
- std::string get_table_name(const std::string& schema,
- const std::string& table);
-
- // Embedding generation (placeholder for GenAI)
- std::vector generate_embedding(const std::string& text,
- const std::string& model);
-
- // Content building strategies
- std::string build_content(const json& row,
- const std::vector& columns,
- const std::string& strategy);
-
+ // Embeddings methods (PLANNED)
+ int create_embedding_tables();
+ int generate_embeddings(int run_id);
+ json search_embeddings(const std::string& query, const std::string& schema = "",
+ const std::string& table = "", int limit = 10);
+
public:
- MySQL_Embeddings(const std::string& path);
- ~MySQL_Embeddings();
-
- int init();
- void close();
-
- // Tool methods
- std::string index_table(...);
- std::string search(...);
- std::string list_indexes();
- std::string delete_index(...);
- std::string reindex(...);
- std::string rebuild_all(...);
-
- bool index_exists(const std::string& schema, const std::string& table);
- SQLite3DB* get_db() { return db; }
-};
-```
-
-### sqlite-rembed Configuration
-
-```cpp
-// Configure rembed client during initialization
-int MySQL_Embeddings::init() {
- // ... open database ...
-
- // Check if mcp rembed client exists
- char* error = NULL;
- std::string check_sql = "SELECT name FROM temp.rembed_clients WHERE name='mcp_embeddings'";
-
- // If not exists, create default client
- // (Requires API key to be configured separately by user)
-
- return 0;
-}
-```
-
-### Vector Insert Example
-
-```cpp
-// Insert embedding with content concatenation
-std::string sql =
- "INSERT INTO embeddings_testdb_orders(rowid, vector, pk_value, metadata) "
- "SELECT "
- " ROWID, "
- " rembed('mcp_embeddings', ?) as vector, "
- " CAST(order_id AS TEXT) as pk_value, "
- " json_object('order_id', order_id, 'customer_name', customer_name) as metadata "
- "FROM testdb.orders "
- "WHERE active = 1";
-
-// Execute with prepared statement
-sqlite3_stmt* stmt;
-db->prepare_v2(sql.c_str(), &stmt);
-(*proxy_sqlite3_bind_text)(stmt, 1, content.c_str(), -1, SQLITE_TRANSIENT);
-SAFE_SQLITE3_STEP2(stmt);
-(*proxy_sqlite3_finalize)(stmt);
-```
-
-### Similarity Search Example
-
-```cpp
-// Generate query embedding
-std::vector query_vec = generate_embedding(query_text, model_name);
-std::string query_vec_json = vector_to_json(query_vec);
-
-// Build search SQL
-std::ostringstream sql;
-sql << "SELECT pk_value, distance, metadata "
- << "FROM embeddings_testdb_orders "
- << "WHERE vector MATCH " << query_vec_json << " "
- << "AND distance < " << min_distance << " "
- << "ORDER BY distance ASC "
- << "LIMIT " << limit;
-
-// Execute and return results
-```
-
-## Configuration Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `mcp-embeddingpath` | `mcp_embeddings.db` | Path to embeddings SQLite database |
-| `mcp-rembed-client` | (none) | Default sqlite-rembed client name (user must configure) |
-
-**sqlite-rembed Configuration** (must be done by user):
-```sql
--- Configure OpenAI client
-INSERT INTO temp.rembed_clients(name, format, model, key)
-VALUES ('mcp_embeddings', 'openai', 'text-embedding-3-small', 'sk-...');
-
--- Or local Ollama
-INSERT INTO temp.rembed_clients(name, format, model, key)
-VALUES ('mcp_embeddings', 'ollama', 'nomic-embed-text', '');
-
--- Or Cohere
-INSERT INTO temp.rembed_clients(name, format, model, key)
-VALUES ('mcp_embeddings', 'cohere', 'embed-english-v3.0', '...');
-```
-
-## Model Support
-
-### Common Embedding Models
-
-| Model | Dimensions | Provider | Format |
-|-------|------------|----------|--------|
-| text-embedding-3-small | 1536 | OpenAI | openai |
-| text-embedding-3-large | 3072 | OpenAI | openai |
-| nomic-embed-text-v1.5 | 768 | Nomic | nomic |
-| all-MiniLM-L6-v2 | 384 | Local (Ollama) | ollama |
-| mxbai-embed-large-v1 | 1024 | MixedBread (Ollama) | ollama |
-
-### Vector Dimension Reference
-
-```cpp
-// Map model names to dimensions
-std::map model_dimensions = {
- {"text-embedding-3-small", 1536},
- {"text-embedding-3-large", 3072},
- {"nomic-embed-text-v1.5", 768},
- {"all-MiniLM-L6-v2", 384},
- {"mxbai-embed-large-v1", 1024}
+ // Embeddings to be maintained during:
+ // - Object processing (static harvest)
+ // - LLM artifact creation
+ // - Catalog rebuild operations
};
```
-## Agent Workflow Examples
-
-### Example 1: Semantic Search
+## Agent Workflow Example (PLANNED)
```python
-# Agent finds semantically similar content
-embed_results = call_tool("embed_search", {
- "query": "customer unhappy with shipping delay",
+# Agent performs semantic search
+semantic_results = call_tool("embedding_search", {
+ "query": "find tables related to customer purchases",
"limit": 10
})
-# Extract primary keys
-order_ids = [r["primary_key_value"] for r in embed_results["results"]]
-
-# Query MySQL for full data
-full_orders = call_tool("run_sql_readonly", {
- "sql": f"SELECT * FROM orders WHERE order_id IN ({','.join(order_ids)})"
-})
-```
-
-### Example 2: Combined FTS + Embeddings
-
-```python
-# FTS for exact keyword match
-keyword_results = call_tool("fts_search", {
- "query": "refund request",
- "limit": 50
+# Agent combines with FTS results
+fts_results = call_tool("catalog_search", {
+ "query": "customer order"
})
-# Embeddings for semantic similarity
-semantic_results = call_tool("embed_search", {
- "query": "customer wants money back",
- "limit": 50
-})
-
-# Combine and deduplicate for best results
-all_ids = set(
- [r["primary_key_value"] for r in keyword_results["results"]] +
- [r["primary_key_value"] for r in semantic_results["results"]]
-)
-```
-
-### Example 3: RAG (Retrieval Augmented Generation)
-
-```python
-# 1. Search for relevant documents
-docs = call_tool("embed_search", {
- "query": user_question,
- "table": "knowledge_base",
- "limit": 5
-})
-
-# 2. Build context from retrieved documents
-context = "\n".join([d["metadata"]["content"] for d in docs["results"]])
-
-# 3. Generate answer using context
-answer = call_llm({
- "prompt": f"Context: {context}\n\nQuestion: {user_question}\n\nAnswer:"
-})
-```
-
-## Comparison: FTS vs Embeddings
-
-| Aspect | FTS (fts_*) | Embeddings (embed_*) |
-|--------|-------------|---------------------|
-| **Search Type** | Lexical (keyword matching) | Semantic (similarity matching) |
-| **Query Example** | "urgent order" | "customer complaint about late delivery" |
-| **Technology** | SQLite FTS5 | sqlite-vec |
-| **Storage** | Text content | Vector embeddings (float arrays) |
-| **External API** | None | sqlite-rembed / GenAI module |
-| **Speed** | Very fast | Fast (but API call latency) |
-| **Use Cases** | Exact phrase matching, filters | Similar content, semantic understanding |
-| **Strengths** | Fast, precise, works offline | Finds related content, handles synonyms |
-| **Weaknesses** | Misses semantic matches | Requires API, slower, needs setup |
-
-## Performance Considerations
-
-### Embedding Generation
-- **API Rate Limits**: OpenAI has rate limits (e.g., 3000 RPM)
-- **Batch Processing**: sqlite-rembed doesn't support batching yet
-- **Latency**: Each embedding = 1 HTTP call (50-500ms)
-- **Cost**: OpenAI charges per token (e.g., $0.00002/1K tokens)
-
-### Vector Storage
-- **Storage**: 1536 floats × 4 bytes = ~6KB per embedding
-- **10,000 rows** = ~60MB for embeddings
-- **Memory**: sqlite-vec loads vectors into memory for search
-
-### Search Performance
-- **KNN Search**: O(n × d) where n=rows, d=dimensions
-- **Typical**: < 100ms for 10K rows, < 1s for 1M rows
-- **Limit**: Use LIMIT or `k = ?` constraint (required by vec0)
-
-## Best Practices
-
-### When to Use Embeddings
-- **Semantic search**: Find similar meanings, not just keywords
-- **Content recommendation**: "Users who liked X also liked Y"
-- **Duplicate detection**: Find similar documents
-- **Categorization**: Cluster similar content
-- **RAG**: Retrieve relevant context for LLM
-
-### When to Use FTS
-- **Exact matching**: Log search, code search
-- **Filters**: Combined with WHERE clauses
-- **Speed critical**: Sub-millisecond response needed
-- **Offline**: No external API access
-
-### Column Selection
-- **Choose meaningful columns**: Text that captures semantic meaning
-- **Avoid IDs/numbers**: Order ID, timestamps (low semantic value)
-- **Combine textually**: `title + description + notes`
-- **Preprocess**: Remove HTML, special characters
-
-### Strategy Selection
-- **concat**: Default, works for most use cases
-- **average**: When columns have independent meaning
-- **separate**: When need column-specific similarity
-
-## Testing Checklist
-
-### Basic Functionality
-- [ ] Create embedding index (single table)
-- [ ] Create embedding index with WHERE clause
-- [ ] Create embedding index with average strategy
-- [ ] Search single table
-- [ ] Search across all tables
-- [ ] List indexes
-- [ ] Delete index
-- [ ] Reindex single table
-- [ ] Rebuild all indexes
-
-### Edge Cases
-- [ ] Empty result sets
-- [ ] NULL values in columns
-- [ ] Special characters in text
-- [ ] Very long text (>10K chars)
-- [ ] Non-ASCII text (Unicode)
-- [ ] API rate limiting
-- [ ] API errors
-- [ ] Invalid model names
-
-### Integration
-- [ ] Works alongside FTS
-- [ ] Works with catalog
-- [ ] SQLite-vec extension loaded
-- [ ] sqlite-rembed client configured
-- [ ] Cross-table semantic search
-
-## GenAI Module Integration (Future)
-
-### Placeholder Interface
-
-```cpp
-// When GenAI module is merged, replace sqlite-rembed calls
-#ifdef HAVE_GENAI_MODULE
- #include "GenAI_Module.h"
-#endif
-
-std::vector MySQL_Embeddings::generate_embedding(
- const std::string& text,
- const std::string& model
-) {
-#ifdef HAVE_GENAI_MODULE
- // Use GenAI module
- return GenAI_Module::generate_embedding(text, model);
-#else
- // Use sqlite-rembed
- std::string sql = "SELECT rembed('mcp_embeddings', ?) as embedding";
- // ... execute and parse ...
- return parse_vector_from_json(result);
-#endif
-}
-```
-
-### Configuration for GenAI
-
-When GenAI module is available, add configuration variable:
-```sql
-SET mcp-genai-provider='local'; -- or 'openai', 'ollama', etc.
-SET mcp-genai-model='nomic-embed-text-v1.5';
+# Agent uses combined results for comprehensive understanding
```
-## Troubleshooting
+## Future Performance Considerations
-### Common Issues
+1. **Batch Processing**: Generate embeddings in batches for performance
+2. **Model Selection**: Support multiple embedding models with different dimensions
+3. **Caching**: Cache frequently used embeddings
+4. **Indexing**: Use ANN (Approximate Nearest Neighbor) for large vector sets
-**Issue**: "Error: no such table: temp.rembed_clients"
-- **Cause**: sqlite-rembed extension not loaded
-- **Fix**: Ensure sqlite-rembed is compiled and auto-registered
+## Implementation Prerequisites
-**Issue**: "Error: rembed client not found"
-- **Cause**: sqlite-rembed client not configured
-- **Fix**: Run INSERT into temp.rembed_clients
+- [ ] sqlite-vec extension compiled into ProxySQL
+- [ ] sqlite-rembed integration with LLM_Bridge
+- [ ] Configuration variable support
+- [ ] Tool handler integration
-**Issue**: "Error: vector dimension mismatch"
-- **Cause**: Model output doesn't match vec0 table dimensions
-- **Fix**: Ensure vector_dim matches model output
+## Notes
-**Issue**: API rate limit exceeded
-- **Cause**: Too many embedding requests
-- **Fix**: Add delays, batch processing (when available), or use local model
+- Vector embeddings will complement FTS for comprehensive search
+- Integration with existing catalog for unified search experience
+- Support for multiple embedding models and providers
+- Automatic embedding generation during discovery processes
-## Notes
+## Version
-- Follow existing patterns from `MySQL_FTS` and `MySQL_Catalog` for SQLite management
-- Use SQLite3DB read-write locks for thread safety
-- Return JSON responses using nlohmann/json library
-- Handle NULL values properly (use empty string as in execute_query)
-- Use prepared statements for SQL safety
-- Log errors using `proxy_error()` and info using `proxy_info()`
-- Table name sanitization: replace `.` and special chars with `_`
-- Always use LIMIT or `k = ?` in vec0 KNN queries (sqlite-vec requirement)
-- Configure sqlite-rembed client before indexing
-- Consider API costs and rate limits when planning bulk indexing
+- **Last Updated:** 2026-01-19
+- **Status:** Planned feature, not yet implemented
diff --git a/doc/Two_Phase_Discovery_Implementation.md b/doc/Two_Phase_Discovery_Implementation.md
new file mode 100644
index 0000000000..233dbae0ea
--- /dev/null
+++ b/doc/Two_Phase_Discovery_Implementation.md
@@ -0,0 +1,337 @@
+# Two-Phase Schema Discovery Redesign - Implementation Summary
+
+## Overview
+
+This document summarizes the implementation of the two-phase schema discovery redesign for ProxySQL MCP. The implementation transforms the previous LLM-only auto-discovery into a **two-phase architecture**:
+
+1. **Phase 1: Static/Auto Discovery** - Deterministic harvest from MySQL INFORMATION_SCHEMA
+2. **Phase 2: LLM Agent Discovery** - Semantic analysis using MCP tools only (NO file I/O)
+
+## Implementation Date
+
+January 17, 2026
+
+## Files Created
+
+### Core Discovery Components
+
+| File | Purpose |
+|------|---------|
+| `include/Discovery_Schema.h` | New catalog schema interface with deterministic + LLM layers |
+| `lib/Discovery_Schema.cpp` | Schema initialization with 20+ tables (runs, objects, columns, indexes, fks, profiles, FTS, LLM artifacts) |
+| `include/Static_Harvester.h` | Static harvester interface for deterministic metadata extraction |
+| `lib/Static_Harvester.cpp` | Deterministic metadata harvest from INFORMATION_SCHEMA (mirrors Python PoC) |
+| `include/Query_Tool_Handler.h` | **REFACTORED**: Now uses Discovery_Schema directly, includes 17 discovery tools |
+| `lib/Query_Tool_Handler.cpp` | **REFACTORED**: All query + discovery tools in unified handler |
+
+### Prompt Files
+
+| File | Purpose |
+|------|---------|
+| `scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/two_phase_discovery_prompt.md` | System prompt for LLM agent (staged discovery, MCP-only I/O) |
+| `scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/two_phase_user_prompt.md` | User prompt with discovery procedure |
+| `scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py` | Orchestration script wrapper for Claude Code |
+
+## Files Modified
+
+| File | Changes |
+|------|--------|
+| `include/Query_Tool_Handler.h` | **COMPLETELY REWRITTEN**: Now uses Discovery_Schema directly, includes MySQL connection pool |
+| `lib/Query_Tool_Handler.cpp` | **COMPLETELY REWRITTEN**: 37 tools (20 original + 17 discovery), direct catalog/harvester usage |
+| `lib/ProxySQL_MCP_Server.cpp` | Updated Query_Tool_Handler initialization (new constructor signature), removed Discovery_Tool_Handler |
+| `include/MCP_Thread.h` | Removed Discovery_Tool_Handler forward declaration and pointer |
+| `lib/Makefile` | Added Discovery_Schema.oo, Static_Harvester.oo (removed Discovery_Tool_Handler.oo) |
+
+## Files Deleted
+
+| File | Reason |
+|------|--------|
+| `include/Discovery_Tool_Handler.h` | Consolidated into Query_Tool_Handler |
+| `lib/Discovery_Tool_Handler.cpp` | Consolidated into Query_Tool_Handler |
+
+## Architecture
+
+**IMPORTANT ARCHITECTURAL NOTE:** All discovery tools are now available through the `/mcp/query` endpoint. The separate `/mcp/discovery` endpoint approach was **removed** in favor of consolidation. Query_Tool_Handler now:
+
+1. Uses `Discovery_Schema` directly (instead of wrapping `MySQL_Tool_Handler`)
+2. Includes MySQL connection pool for direct queries
+3. Provides all 37 tools (20 original + 17 discovery) through a single endpoint
+
+### Phase 1: Static Discovery (C++)
+
+The `Static_Harvester` class performs deterministic metadata extraction:
+
+```
+MySQL INFORMATION_SCHEMA → Static_Harvester → Discovery_Schema SQLite
+```
+
+**Harvest stages:**
+1. Schemas (`information_schema.SCHEMATA`)
+2. Objects (`information_schema.TABLES`, `ROUTINES`)
+3. Columns (`information_schema.COLUMNS`) with derived hints (is_time, is_id_like)
+4. Indexes (`information_schema.STATISTICS`)
+5. Foreign Keys (`KEY_COLUMN_USAGE`, `REFERENTIAL_CONSTRAINTS`)
+6. View definitions (`information_schema.VIEWS`)
+7. Quick profiles (metadata-based analysis)
+8. FTS5 index rebuild
+
+**Derived field calculations:**
+| Field | Calculation |
+|-------|-------------|
+| `is_time` | `data_type IN ('date','datetime','timestamp','time','year')` |
+| `is_id_like` | `column_name REGEXP '(^id$|_id$)'` |
+| `has_primary_key` | `EXISTS (SELECT 1 FROM indexes WHERE is_primary=1)` |
+| `has_foreign_keys` | `EXISTS (SELECT 1 FROM foreign_keys WHERE child_object_id=?)` |
+| `has_time_column` | `EXISTS (SELECT 1 FROM columns WHERE is_time=1)` |
+
+### Phase 2: LLM Agent Discovery (MCP Tools)
+
+The LLM agent (via Claude Code) performs semantic analysis using 18+ MCP tools:
+
+**Discovery Trigger (1 tool):**
+- `discovery.run_static` - Triggers ProxySQL's static harvest
+
+**Catalog Tools (5 tools):**
+- `catalog.init` - Initialize/migrate SQLite schema
+- `catalog.search` - FTS5 search over objects
+- `catalog.get_object` - Get object with columns/indexes/FKs
+- `catalog.list_objects` - List objects (paged)
+- `catalog.get_relationships` - Get FKs, view deps, inferred relationships
+
+**Agent Tools (3 tools):**
+- `agent.run_start` - Create agent run bound to run_id
+- `agent.run_finish` - Mark agent run success/failed
+- `agent.event_append` - Log tool calls, results, decisions
+
+**LLM Memory Tools (9 tools):**
+- `llm.summary_upsert` - Store semantic summary for object
+- `llm.summary_get` - Get semantic summary
+- `llm.relationship_upsert` - Store inferred relationship
+- `llm.domain_upsert` - Create/update domain
+- `llm.domain_set_members` - Set domain members
+- `llm.metric_upsert` - Store metric definition
+- `llm.question_template_add` - Add question template
+- `llm.note_add` - Add durable note
+- `llm.search` - FTS over LLM artifacts
+
+## Database Schema
+
+### Deterministic Layer Tables
+
+| Table | Purpose |
+|-------|---------|
+| `runs` | Track each discovery run (run_id, started_at, finished_at, source_dsn, mysql_version) |
+| `schemas` | Discovered MySQL schemas (schema_name, charset, collation) |
+| `objects` | Tables/views/routines/triggers with metadata (engine, rows_est, has_pk, has_fks, has_time) |
+| `columns` | Column details (data_type, is_nullable, is_pk, is_unique, is_indexed, is_time, is_id_like) |
+| `indexes` | Index metadata (is_unique, is_primary, index_type, cardinality) |
+| `index_columns` | Ordered index columns |
+| `foreign_keys` | FK relationships |
+| `foreign_key_columns` | Ordered FK columns |
+| `profiles` | Profiling results (JSON for extensibility) |
+| `fts_objects` | FTS5 index over objects (contentless) |
+
+### LLM Agent Layer Tables
+
+| Table | Purpose |
+|-------|---------|
+| `agent_runs` | LLM agent runs (bound to deterministic run_id) |
+| `agent_events` | Tool calls, results, decisions (traceability) |
+| `llm_object_summaries` | Per-object semantic summaries (hypothesis, grain, dims/measures, joins) |
+| `llm_relationships` | LLM-inferred relationships with confidence |
+| `llm_domains` | Domain clusters (billing, sales, auth, etc.) |
+| `llm_domain_members` | Object-to-domain mapping with roles |
+| `llm_metrics` | Metric/KPI definitions |
+| `llm_question_templates` | NL → structured query plan mappings |
+| `llm_notes` | Free-form durable notes |
+| `fts_llm` | FTS5 over LLM artifacts |
+
+## Usage
+
+The two-phase discovery provides two ways to discover your database schema:
+
+### Phase 1: Static Harvest (Direct curl)
+
+Phase 1 is a simple HTTP POST to trigger deterministic metadata extraction. No Claude Code required.
+
+```bash
+# Option A: Using the convenience script (recommended)
+cd scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/
+./static_harvest.sh --schema sales --notes "Production sales database discovery"
+
+# Option B: Using curl directly
+curl -k -X POST https://localhost:6071/mcp/query \
+ -H "Content-Type: application/json" \
+ -d '{
+ "jsonrpc": "2.0",
+ "id": 1,
+ "method": "tools/call",
+ "params": {
+ "name": "discovery.run_static",
+ "arguments": {
+ "schema_filter": "sales",
+ "notes": "Production sales database discovery"
+ }
+ }
+ }'
+# Returns: { run_id: 1, started_at: "...", objects_count: 45, columns_count: 380 }
+```
+
+### Phase 2: LLM Agent Discovery (via two_phase_discovery.py)
+
+Phase 2 uses Claude Code for semantic analysis. Requires MCP configuration.
+
+```bash
+# Step 1: Copy example MCP config and customize
+cp scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/mcp_config.example.json mcp_config.json
+# Edit mcp_config.json to set your PROXYSQL_MCP_ENDPOINT if needed
+
+# Step 2: Run the two-phase discovery
+./scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py \
+ --mcp-config mcp_config.json \
+ --schema sales \
+ --model claude-3.5-sonnet
+
+# Dry-run mode (preview without executing)
+./scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py \
+ --mcp-config mcp_config.json \
+ --schema test \
+ --dry-run
+```
+
+### Direct MCP Tool Calls (via /mcp/query endpoint)
+
+You can also call discovery tools directly via the MCP endpoint:
+
+```bash
+# All discovery tools are available via /mcp/query endpoint
+curl -k -X POST https://localhost:6071/mcp/query \
+ -H "Content-Type: application/json" \
+ -d '{
+ "jsonrpc": "2.0",
+ "id": 1,
+ "method": "tools/call",
+ "params": {
+ "name": "discovery.run_static",
+ "arguments": {
+ "schema_filter": "sales",
+ "notes": "Production sales database discovery"
+ }
+ }
+ }'
+# Returns: { run_id: 1, started_at: "...", objects_count: 45, columns_count: 380 }
+
+# Phase 2: LLM agent discovery
+curl -k -X POST https://localhost:6071/mcp/query \
+ -H "Content-Type: application/json" \
+ -d '{
+ "jsonrpc": "2.0",
+ "id": 2,
+ "method": "tools/call",
+ "params": {
+ "name": "agent.run_start",
+ "arguments": {
+ "run_id": 1,
+ "model_name": "claude-3.5-sonnet"
+ }
+ }
+ }'
+# Returns: { agent_run_id: 1 }
+```
+
+## Discovery Workflow
+
+```
+Stage 0: Start and plan
+├─> discovery.run_static() → run_id
+├─> agent.run_start(run_id) → agent_run_id
+└─> agent.event_append(plan, budgets)
+
+Stage 1: Triage and prioritization
+└─> catalog.list_objects() + catalog.search() → build prioritized backlog
+
+Stage 2: Per-object semantic summarization
+└─> catalog.get_object() + catalog.get_relationships()
+ └─> llm.summary_upsert() (50+ high-value objects)
+
+Stage 3: Relationship enhancement
+└─> llm.relationship_upsert() (where FKs missing or unclear)
+
+Stage 4: Domain clustering and synthesis
+└─> llm.domain_upsert() + llm.domain_set_members()
+ └─> llm.note_add(domain descriptions)
+
+Stage 5: "Answerability" artifacts
+├─> llm.metric_upsert() (10-30 metrics)
+└─> llm.question_template_add() (15-50 question templates)
+
+Shutdown:
+├─> agent.event_append(final_summary)
+└─> agent.run_finish(success)
+```
+
+## Quality Rules
+
+Confidence scores:
+- **0.9–1.0**: supported by schema + constraints or very strong evidence
+- **0.6–0.8**: likely, supported by multiple signals but not guaranteed
+- **0.3–0.5**: tentative hypothesis; mark warnings and what's needed to confirm
+
+## Critical Constraint: NO FILES
+
+- LLM agent MUST NOT create/read/modify any local files
+- All outputs MUST be persisted exclusively via MCP tools
+- Use `agent_events` and `llm_notes` as scratchpad
+
+## Verification
+
+To verify the implementation:
+
+```bash
+# Build ProxySQL
+cd /home/rene/proxysql-vec
+make -j$(nproc)
+
+# Verify new discovery components exist
+ls -la include/Discovery_Schema.h include/Static_Harvester.h
+ls -la lib/Discovery_Schema.cpp lib/Static_Harvester.cpp
+
+# Verify Discovery_Tool_Handler was removed (should return nothing)
+ls include/Discovery_Tool_Handler.h 2>&1 # Should fail
+ls lib/Discovery_Tool_Handler.cpp 2>&1 # Should fail
+
+# Verify Query_Tool_Handler uses Discovery_Schema
+grep -n "Discovery_Schema" include/Query_Tool_Handler.h
+grep -n "Static_Harvester" include/Query_Tool_Handler.h
+
+# Verify Query_Tool_Handler has discovery tools
+grep -n "discovery.run_static" lib/Query_Tool_Handler.cpp
+grep -n "agent.run_start" lib/Query_Tool_Handler.cpp
+grep -n "llm.summary_upsert" lib/Query_Tool_Handler.cpp
+
+# Test Phase 1 (curl)
+curl -k -X POST https://localhost:6071/mcp/query \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"discovery.run_static","arguments":{"schema_filter":"test"}}}'
+# Should return: { run_id: 1, objects_count: X, columns_count: Y }
+
+# Test Phase 2 (two_phase_discovery.py)
+cd scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/
+cp mcp_config.example.json mcp_config.json
+./two_phase_discovery.py --dry-run --mcp-config mcp_config.json --schema test
+```
+
+## Next Steps
+
+1. **Build and test**: Compile ProxySQL and test with a small database
+2. **Integration testing**: Test with medium database (100+ tables)
+3. **Documentation updates**: Update main README and MCP docs
+4. **Migration guide**: Document transition from legacy 6-agent to new two-phase system
+
+## References
+
+- Python PoC: `/tmp/mysql_autodiscovery_poc.py`
+- Schema specification: `/tmp/schema.sql`
+- MCP tools specification: `/tmp/mcp_tools_discovery_catalog.json`
+- System prompt reference: `/tmp/system_prompt.md`
+- User prompt reference: `/tmp/user_prompt.md`
diff --git a/doc/rag-documentation.md b/doc/rag-documentation.md
new file mode 100644
index 0000000000..61c9cbaad7
--- /dev/null
+++ b/doc/rag-documentation.md
@@ -0,0 +1,149 @@
+# RAG (Retrieval-Augmented Generation) in ProxySQL
+
+## Overview
+
+ProxySQL's RAG subsystem provides retrieval capabilities for LLM-powered applications. It allows you to:
+
+- Store documents and their embeddings in a SQLite-based vector database
+- Perform keyword search (FTS), semantic search (vector), and hybrid search
+- Fetch document and chunk content
+- Refetch authoritative data from source databases
+- Monitor RAG system statistics
+
+## Configuration
+
+To enable RAG functionality, you need to enable the GenAI module and RAG features:
+
+```sql
+-- Enable GenAI module
+SET genai.enabled = true;
+
+-- Enable RAG features
+SET genai.rag_enabled = true;
+
+-- Configure RAG parameters (optional)
+SET genai.rag_k_max = 50;
+SET genai.rag_candidates_max = 500;
+SET genai.rag_timeout_ms = 2000;
+```
+
+## Available MCP Tools
+
+The RAG subsystem provides the following MCP tools via the `/mcp/rag` endpoint:
+
+### Search Tools
+
+1. **rag.search_fts** - Keyword search using FTS5
+ ```json
+ {
+ "query": "search terms",
+ "k": 10
+ }
+ ```
+
+2. **rag.search_vector** - Semantic search using vector embeddings
+ ```json
+ {
+ "query_text": "semantic search query",
+ "k": 10
+ }
+ ```
+
+3. **rag.search_hybrid** - Hybrid search combining FTS and vectors
+ ```json
+ {
+ "query": "search query",
+ "mode": "fuse", // or "fts_then_vec"
+ "k": 10
+ }
+ ```
+
+### Fetch Tools
+
+4. **rag.get_chunks** - Fetch chunk content by chunk_id
+ ```json
+ {
+ "chunk_ids": ["chunk1", "chunk2"],
+ "return": {
+ "include_title": true,
+ "include_doc_metadata": true,
+ "include_chunk_metadata": true
+ }
+ }
+ ```
+
+5. **rag.get_docs** - Fetch document content by doc_id
+ ```json
+ {
+ "doc_ids": ["doc1", "doc2"],
+ "return": {
+ "include_body": true,
+ "include_metadata": true
+ }
+ }
+ ```
+
+6. **rag.fetch_from_source** - Refetch authoritative data from source database
+ ```json
+ {
+ "doc_ids": ["doc1"],
+ "columns": ["Id", "Title", "Body"],
+ "limits": {
+ "max_rows": 10,
+ "max_bytes": 200000
+ }
+ }
+ ```
+
+### Admin Tools
+
+7. **rag.admin.stats** - Get operational statistics for RAG system
+ ```json
+ {}
+ ```
+
+## Database Schema
+
+The RAG subsystem uses the following tables in the vector database (`/var/lib/proxysql/ai_features.db`):
+
+- **rag_sources** - Control plane for ingestion configuration
+- **rag_documents** - Canonical documents
+- **rag_chunks** - Retrieval units (chunked content)
+- **rag_fts_chunks** - FTS5 index for keyword search
+- **rag_vec_chunks** - Vector index for semantic search
+- **rag_sync_state** - Sync state for incremental ingestion
+- **rag_chunk_view** - Convenience view for debugging
+
+## Testing
+
+You can test the RAG functionality using the provided test scripts:
+
+```bash
+# Test RAG functionality via MCP endpoint
+./scripts/mcp/test_rag.sh
+
+# Test RAG database schema
+cd test/rag
+make test_rag_schema
+./test_rag_schema
+```
+
+## Security
+
+The RAG subsystem includes several security features:
+
+- Input validation and sanitization
+- Query length limits
+- Result size limits
+- Timeouts for all operations
+- Column whitelisting for refetch operations
+- Row and byte limits for all operations
+
+## Performance
+
+Recommended performance settings:
+
+- Set appropriate timeouts (250-2000ms)
+- Limit result sizes (k_max=50, candidates_max=500)
+- Use connection pooling for source database connections
+- Monitor resource usage and adjust limits accordingly
\ No newline at end of file
diff --git a/doc/rag-doxygen-documentation-summary.md b/doc/rag-doxygen-documentation-summary.md
new file mode 100644
index 0000000000..75042f6e0c
--- /dev/null
+++ b/doc/rag-doxygen-documentation-summary.md
@@ -0,0 +1,161 @@
+# RAG Subsystem Doxygen Documentation Summary
+
+## Overview
+
+This document provides a summary of the Doxygen documentation added to the RAG (Retrieval-Augmented Generation) subsystem in ProxySQL. The documentation follows standard Doxygen conventions with inline comments in the source code files.
+
+## Documented Files
+
+### 1. Header File
+- **File**: `include/RAG_Tool_Handler.h`
+- **Documentation**: Comprehensive class and method documentation with detailed parameter descriptions, return values, and cross-references.
+
+### 2. Implementation File
+- **File**: `lib/RAG_Tool_Handler.cpp`
+- **Documentation**: Detailed function documentation with implementation-specific notes, parameter descriptions, and cross-references.
+
+## Documentation Structure
+
+### Class Documentation
+The `RAG_Tool_Handler` class is thoroughly documented with:
+- **Class overview**: General description of the class purpose and functionality
+- **Group membership**: Categorized under `@ingroup mcp` and `@ingroup rag`
+- **Member variables**: Detailed documentation of all private members with `///` comments
+- **Method documentation**: Complete documentation for all public and private methods
+
+### Method Documentation
+Each method includes:
+- **Brief description**: Concise summary of the method's purpose
+- **Detailed description**: Comprehensive explanation of functionality
+- **Parameters**: Detailed description of each parameter with `@param` tags
+- **Return values**: Description of return values with `@return` tags
+- **Error conditions**: Documentation of possible error scenarios
+- **Cross-references**: Links to related methods with `@see` tags
+- **Implementation notes**: Special considerations or implementation details
+
+### Helper Functions
+Helper functions are documented with:
+- **Purpose**: Clear explanation of what the function does
+- **Parameter handling**: Details on how parameters are processed
+- **Error handling**: Documentation of error conditions and recovery
+- **Usage examples**: References to where the function is used
+
+## Key Documentation Features
+
+### 1. Configuration Parameters
+All configuration parameters are documented with:
+- Default values
+- Valid ranges
+- Usage examples
+- Related configuration options
+
+### 2. Tool Specifications
+Each RAG tool is documented with:
+- **Input parameters**: Complete schema with types and descriptions
+- **Output format**: Response structure documentation
+- **Error handling**: Possible error responses
+- **Usage examples**: Common use cases
+
+### 3. Security Features
+Security-related functionality is documented with:
+- **Input validation**: Parameter validation rules
+- **Limits and constraints**: Resource limits and constraints
+- **Error handling**: Security-related error conditions
+
+### 4. Performance Considerations
+Performance-related aspects are documented with:
+- **Optimization strategies**: Performance optimization techniques used
+- **Resource management**: Memory and connection management
+- **Scalability considerations**: Scalability features and limitations
+
+## Documentation Tags Used
+
+### Standard Doxygen Tags
+- `@file`: File description
+- `@brief`: Brief description
+- `@param`: Parameter description
+- `@return`: Return value description
+- `@see`: Cross-reference to related items
+- `@ingroup`: Group membership
+- `@author`: Author information
+- `@date`: File creation/update date
+- `@copyright`: Copyright information
+
+### Specialized Tags
+- `@defgroup`: Group definition
+- `@addtogroup`: Group membership
+- `@exception`: Exception documentation
+- `@note`: Additional notes
+- `@warning`: Warning information
+- `@todo`: Future work items
+
+## Usage Instructions
+
+### Generating Documentation
+To generate the Doxygen documentation:
+
+```bash
+# Install Doxygen (if not already installed)
+sudo apt-get install doxygen graphviz
+
+# Generate documentation
+cd /path/to/proxysql
+doxygen Doxyfile
+```
+
+### Viewing Documentation
+The generated documentation will be available in:
+- **HTML format**: `docs/html/index.html`
+- **LaTeX format**: `docs/latex/refman.tex`
+
+## Documentation Completeness
+
+### Covered Components
+✅ **RAG_Tool_Handler class**: Complete class documentation
+✅ **Constructor/Destructor**: Detailed lifecycle method documentation
+✅ **Public methods**: All public interface methods documented
+✅ **Private methods**: All private helper methods documented
+✅ **Configuration parameters**: All configuration options documented
+✅ **Tool specifications**: All RAG tools documented with schemas
+✅ **Error handling**: Comprehensive error condition documentation
+✅ **Security features**: Security-related functionality documented
+✅ **Performance aspects**: Performance considerations documented
+
+### Documentation Quality
+✅ **Consistency**: Uniform documentation style across all files
+✅ **Completeness**: All public interfaces documented
+✅ **Accuracy**: Documentation matches implementation
+✅ **Clarity**: Clear and concise descriptions
+✅ **Cross-referencing**: Proper links between related components
+✅ **Examples**: Usage examples where appropriate
+
+## Maintenance Guidelines
+
+### Keeping Documentation Updated
+1. **Update with code changes**: Always update documentation when modifying code
+2. **Review regularly**: Periodically review documentation for accuracy
+3. **Test generation**: Verify that documentation generates without warnings
+4. **Cross-reference updates**: Update cross-references when adding new methods
+
+### Documentation Standards
+1. **Consistent formatting**: Follow established documentation patterns
+2. **Clear language**: Use simple, precise language
+3. **Complete coverage**: Document all parameters and return values
+4. **Practical examples**: Include relevant usage examples
+5. **Error scenarios**: Document possible error conditions
+
+## Benefits
+
+### For Developers
+- **Easier onboarding**: New developers can quickly understand the codebase
+- **Reduced debugging time**: Clear documentation helps identify issues faster
+- **Better collaboration**: Shared understanding of component interfaces
+- **Code quality**: Documentation encourages better code design
+
+### For Maintenance
+- **Reduced maintenance overhead**: Clear documentation reduces maintenance time
+- **Easier upgrades**: Documentation helps understand impact of changes
+- **Better troubleshooting**: Detailed error documentation aids troubleshooting
+- **Knowledge retention**: Documentation preserves implementation knowledge
+
+The RAG subsystem is now fully documented with comprehensive Doxygen comments that provide clear guidance for developers working with the codebase.
\ No newline at end of file
diff --git a/doc/rag-doxygen-documentation.md b/doc/rag-doxygen-documentation.md
new file mode 100644
index 0000000000..0c1351a17b
--- /dev/null
+++ b/doc/rag-doxygen-documentation.md
@@ -0,0 +1,351 @@
+# RAG Subsystem Doxygen Documentation
+
+## Overview
+
+The RAG (Retrieval-Augmented Generation) subsystem provides a comprehensive set of tools for semantic search and document retrieval through the MCP (Model Context Protocol). This documentation details the Doxygen-style comments added to the RAG implementation.
+
+## Main Classes
+
+### RAG_Tool_Handler
+
+The primary class that implements all RAG functionality through the MCP protocol.
+
+#### Class Definition
+```cpp
+class RAG_Tool_Handler : public MCP_Tool_Handler
+```
+
+#### Constructor
+```cpp
+/**
+ * @brief Constructor
+ * @param ai_mgr Pointer to AI_Features_Manager for database access and configuration
+ *
+ * Initializes the RAG tool handler with configuration parameters from GenAI_Thread
+ * if available, otherwise uses default values.
+ *
+ * Configuration parameters:
+ * - k_max: Maximum number of search results (default: 50)
+ * - candidates_max: Maximum number of candidates for hybrid search (default: 500)
+ * - query_max_bytes: Maximum query length in bytes (default: 8192)
+ * - response_max_bytes: Maximum response size in bytes (default: 5000000)
+ * - timeout_ms: Operation timeout in milliseconds (default: 2000)
+ */
+RAG_Tool_Handler(AI_Features_Manager* ai_mgr);
+```
+
+#### Public Methods
+
+##### get_tool_list()
+```cpp
+/**
+ * @brief Get list of available RAG tools
+ * @return JSON object containing tool definitions and schemas
+ *
+ * Returns a comprehensive list of all available RAG tools with their
+ * input schemas and descriptions. Tools include:
+ * - rag.search_fts: Keyword search using FTS5
+ * - rag.search_vector: Semantic search using vector embeddings
+ * - rag.search_hybrid: Hybrid search combining FTS and vectors
+ * - rag.get_chunks: Fetch chunk content by chunk_id
+ * - rag.get_docs: Fetch document content by doc_id
+ * - rag.fetch_from_source: Refetch authoritative data from source
+ * - rag.admin.stats: Operational statistics
+ */
+json get_tool_list() override;
+```
+
+##### execute_tool()
+```cpp
+/**
+ * @brief Execute a RAG tool with arguments
+ * @param tool_name Name of the tool to execute
+ * @param arguments JSON object containing tool arguments
+ * @return JSON response with results or error information
+ *
+ * Executes the specified RAG tool with the provided arguments. Handles
+ * input validation, parameter processing, database queries, and result
+ * formatting according to MCP specifications.
+ *
+ * Supported tools:
+ * - rag.search_fts: Full-text search over documents
+ * - rag.search_vector: Vector similarity search
+ * - rag.search_hybrid: Hybrid search with two modes (fuse, fts_then_vec)
+ * - rag.get_chunks: Retrieve chunk content by ID
+ * - rag.get_docs: Retrieve document content by ID
+ * - rag.fetch_from_source: Refetch data from authoritative source
+ * - rag.admin.stats: Get operational statistics
+ */
+json execute_tool(const std::string& tool_name, const json& arguments) override;
+```
+
+#### Private Helper Methods
+
+##### Database and Query Helpers
+
+```cpp
+/**
+ * @brief Execute database query and return results
+ * @param query SQL query string to execute
+ * @return SQLite3_result pointer or NULL on error
+ *
+ * Executes a SQL query against the vector database and returns the results.
+ * Handles error checking and logging. The caller is responsible for freeing
+ * the returned SQLite3_result.
+ */
+SQLite3_result* execute_query(const char* query);
+
+/**
+ * @brief Validate and limit k parameter
+ * @param k Requested number of results
+ * @return Validated k value within configured limits
+ *
+ * Ensures the k parameter is within acceptable bounds (1 to k_max).
+ * Returns default value of 10 if k is invalid.
+ */
+int validate_k(int k);
+
+/**
+ * @brief Validate and limit candidates parameter
+ * @param candidates Requested number of candidates
+ * @return Validated candidates value within configured limits
+ *
+ * Ensures the candidates parameter is within acceptable bounds (1 to candidates_max).
+ * Returns default value of 50 if candidates is invalid.
+ */
+int validate_candidates(int candidates);
+
+/**
+ * @brief Validate query length
+ * @param query Query string to validate
+ * @return true if query is within length limits, false otherwise
+ *
+ * Checks if the query string length is within the configured query_max_bytes limit.
+ */
+bool validate_query_length(const std::string& query);
+```
+
+##### JSON Parameter Extraction
+
+```cpp
+/**
+ * @brief Extract string parameter from JSON
+ * @param j JSON object to extract from
+ * @param key Parameter key to extract
+ * @param default_val Default value if key not found
+ * @return Extracted string value or default
+ *
+ * Safely extracts a string parameter from a JSON object, handling type
+ * conversion if necessary. Returns the default value if the key is not
+ * found or cannot be converted to a string.
+ */
+static std::string get_json_string(const json& j, const std::string& key,
+ const std::string& default_val = "");
+
+/**
+ * @brief Extract int parameter from JSON
+ * @param j JSON object to extract from
+ * @param key Parameter key to extract
+ * @param default_val Default value if key not found
+ * @return Extracted int value or default
+ *
+ * Safely extracts an integer parameter from a JSON object, handling type
+ * conversion from string if necessary. Returns the default value if the
+ * key is not found or cannot be converted to an integer.
+ */
+static int get_json_int(const json& j, const std::string& key, int default_val = 0);
+
+/**
+ * @brief Extract bool parameter from JSON
+ * @param j JSON object to extract from
+ * @param key Parameter key to extract
+ * @param default_val Default value if key not found
+ * @return Extracted bool value or default
+ *
+ * Safely extracts a boolean parameter from a JSON object, handling type
+ * conversion from string or integer if necessary. Returns the default
+ * value if the key is not found or cannot be converted to a boolean.
+ */
+static bool get_json_bool(const json& j, const std::string& key, bool default_val = false);
+
+/**
+ * @brief Extract string array from JSON
+ * @param j JSON object to extract from
+ * @param key Parameter key to extract
+ * @return Vector of extracted strings
+ *
+ * Safely extracts a string array parameter from a JSON object, filtering
+ * out non-string elements. Returns an empty vector if the key is not
+ * found or is not an array.
+ */
+static std::vector get_json_string_array(const json& j, const std::string& key);
+
+/**
+ * @brief Extract int array from JSON
+ * @param j JSON object to extract from
+ * @param key Parameter key to extract
+ * @return Vector of extracted integers
+ *
+ * Safely extracts an integer array parameter from a JSON object, handling
+ * type conversion from string if necessary. Returns an empty vector if
+ * the key is not found or is not an array.
+ */
+static std::vector get_json_int_array(const json& j, const std::string& key);
+```
+
+##### Scoring and Normalization
+
+```cpp
+/**
+ * @brief Compute Reciprocal Rank Fusion score
+ * @param rank Rank position (1-based)
+ * @param k0 Smoothing parameter
+ * @param weight Weight factor for this ranking
+ * @return RRF score
+ *
+ * Computes the Reciprocal Rank Fusion score for hybrid search ranking.
+ * Formula: weight / (k0 + rank)
+ */
+double compute_rrf_score(int rank, int k0, double weight);
+
+/**
+ * @brief Normalize scores to 0-1 range (higher is better)
+ * @param score Raw score to normalize
+ * @param score_type Type of score being normalized
+ * @return Normalized score in 0-1 range
+ *
+ * Normalizes various types of scores to a consistent 0-1 range where
+ * higher values indicate better matches. Different score types may
+ * require different normalization approaches.
+ */
+double normalize_score(double score, const std::string& score_type);
+```
+
+## Tool Specifications
+
+### rag.search_fts
+Keyword search over documents using FTS5.
+
+#### Parameters
+- `query` (string, required): Search query string
+- `k` (integer): Number of results to return (default: 10, max: 50)
+- `offset` (integer): Offset for pagination (default: 0)
+- `filters` (object): Filter criteria for results
+- `return` (object): Return options for result fields
+
+#### Filters
+- `source_ids` (array of integers): Filter by source IDs
+- `source_names` (array of strings): Filter by source names
+- `doc_ids` (array of strings): Filter by document IDs
+- `min_score` (number): Minimum score threshold
+- `post_type_ids` (array of integers): Filter by post type IDs
+- `tags_any` (array of strings): Filter by any of these tags
+- `tags_all` (array of strings): Filter by all of these tags
+- `created_after` (string): Filter by creation date (after)
+- `created_before` (string): Filter by creation date (before)
+
+#### Return Options
+- `include_title` (boolean): Include title in results (default: true)
+- `include_metadata` (boolean): Include metadata in results (default: true)
+- `include_snippets` (boolean): Include snippets in results (default: false)
+
+### rag.search_vector
+Semantic search over documents using vector embeddings.
+
+#### Parameters
+- `query_text` (string, required): Text to search semantically
+- `k` (integer): Number of results to return (default: 10, max: 50)
+- `filters` (object): Filter criteria for results
+- `embedding` (object): Embedding model specification
+- `query_embedding` (object): Precomputed query embedding
+- `return` (object): Return options for result fields
+
+### rag.search_hybrid
+Hybrid search combining FTS and vector search.
+
+#### Parameters
+- `query` (string, required): Search query for both FTS and vector
+- `k` (integer): Number of results to return (default: 10, max: 50)
+- `mode` (string): Search mode: 'fuse' or 'fts_then_vec'
+- `filters` (object): Filter criteria for results
+- `fuse` (object): Parameters for fuse mode
+- `fts_then_vec` (object): Parameters for fts_then_vec mode
+
+#### Fuse Mode Parameters
+- `fts_k` (integer): Number of FTS results for fusion (default: 50)
+- `vec_k` (integer): Number of vector results for fusion (default: 50)
+- `rrf_k0` (integer): RRF smoothing parameter (default: 60)
+- `w_fts` (number): Weight for FTS scores (default: 1.0)
+- `w_vec` (number): Weight for vector scores (default: 1.0)
+
+#### FTS Then Vector Mode Parameters
+- `candidates_k` (integer): FTS candidates to generate (default: 200)
+- `rerank_k` (integer): Candidates to rerank with vector search (default: 50)
+- `vec_metric` (string): Vector similarity metric (default: 'cosine')
+
+### rag.get_chunks
+Fetch chunk content by chunk_id.
+
+#### Parameters
+- `chunk_ids` (array of strings, required): List of chunk IDs to fetch
+- `return` (object): Return options for result fields
+
+### rag.get_docs
+Fetch document content by doc_id.
+
+#### Parameters
+- `doc_ids` (array of strings, required): List of document IDs to fetch
+- `return` (object): Return options for result fields
+
+### rag.fetch_from_source
+Refetch authoritative data from source database.
+
+#### Parameters
+- `doc_ids` (array of strings, required): List of document IDs to refetch
+- `columns` (array of strings): List of columns to fetch
+- `limits` (object): Limits for the fetch operation
+
+### rag.admin.stats
+Get operational statistics for RAG system.
+
+#### Parameters
+None
+
+## Database Schema
+
+The RAG subsystem uses the following tables in the vector database:
+
+1. `rag_sources`: Ingestion configuration and source metadata
+2. `rag_documents`: Canonical documents with stable IDs
+3. `rag_chunks`: Chunked content for retrieval
+4. `rag_fts_chunks`: FTS5 contentless index for keyword search
+5. `rag_vec_chunks`: sqlite3-vec virtual table for vector similarity search
+6. `rag_sync_state`: Sync state tracking for incremental ingestion
+7. `rag_chunk_view`: Convenience view for debugging
+
+## Security Features
+
+1. **Input Validation**: Strict validation of all parameters and filters
+2. **Query Limits**: Maximum limits on query length, result count, and candidates
+3. **Timeouts**: Configurable operation timeouts to prevent resource exhaustion
+4. **Column Whitelisting**: Strict column filtering for refetch operations
+5. **Row and Byte Limits**: Maximum limits on returned data size
+6. **Parameter Binding**: Safe parameter binding to prevent SQL injection
+
+## Performance Features
+
+1. **Prepared Statements**: Efficient query execution with prepared statements
+2. **Connection Management**: Proper database connection handling
+3. **SQLite3-vec Integration**: Optimized vector operations
+4. **FTS5 Integration**: Efficient full-text search capabilities
+5. **Indexing Strategies**: Proper database indexing for performance
+6. **Result Caching**: Efficient result processing and formatting
+
+## Configuration Variables
+
+1. `genai_rag_enabled`: Enable RAG features
+2. `genai_rag_k_max`: Maximum k for search results (default: 50)
+3. `genai_rag_candidates_max`: Maximum candidates for hybrid search (default: 500)
+4. `genai_rag_query_max_bytes`: Maximum query length in bytes (default: 8192)
+5. `genai_rag_response_max_bytes`: Maximum response size in bytes (default: 5000000)
+6. `genai_rag_timeout_ms`: RAG operation timeout in ms (default: 2000)
\ No newline at end of file
diff --git a/doc/rag-examples.md b/doc/rag-examples.md
new file mode 100644
index 0000000000..8acb913ff5
--- /dev/null
+++ b/doc/rag-examples.md
@@ -0,0 +1,94 @@
+# RAG Tool Examples
+
+This document provides examples of how to use the RAG tools via the MCP endpoint.
+
+## Prerequisites
+
+Make sure ProxySQL is running with GenAI and RAG enabled:
+
+```sql
+-- In ProxySQL admin interface
+SET genai.enabled = true;
+SET genai.rag_enabled = true;
+LOAD genai VARIABLES TO RUNTIME;
+```
+
+## Tool Discovery
+
+### List all RAG tools
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/list","id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+### Get tool description
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/describe","params":{"name":"rag.search_fts"},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+## Search Tools
+
+### FTS Search
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"rag.search_fts","arguments":{"query":"mysql performance","k":5}},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+### Vector Search
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"rag.search_vector","arguments":{"query_text":"database optimization techniques","k":5}},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+### Hybrid Search
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"rag.search_hybrid","arguments":{"query":"sql query optimization","mode":"fuse","k":5}},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+## Fetch Tools
+
+### Get Chunks
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"rag.get_chunks","arguments":{"chunk_ids":["chunk1","chunk2"]}},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+### Get Documents
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"rag.get_docs","arguments":{"doc_ids":["doc1","doc2"]}},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
+
+## Admin Tools
+
+### Get Statistics
+
+```bash
+curl -k -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"rag.admin.stats"},"id":"1"}' \
+ https://127.0.0.1:6071/mcp/rag
+```
\ No newline at end of file
diff --git a/include/Discovery_Schema.h b/include/Discovery_Schema.h
new file mode 100644
index 0000000000..a8d9400df4
--- /dev/null
+++ b/include/Discovery_Schema.h
@@ -0,0 +1,884 @@
+#ifndef CLASS_DISCOVERY_SCHEMA_H
+#define CLASS_DISCOVERY_SCHEMA_H
+
+#include "sqlite3db.h"
+#include
+#include
+#include
+#include
+#include
+#include "json.hpp"
+
+/**
+ * @brief MCP query rule structure
+ *
+ * Action is inferred from rule properties:
+ * - if error_msg != NULL → block
+ * - if replace_pattern != NULL → rewrite
+ * - if timeout_ms > 0 → timeout
+ * - otherwise → allow
+ *
+ * Note: 'hits' is only for in-memory tracking, not persisted to the table.
+ */
+struct MCP_Query_Rule {
+ int rule_id;
+ bool active;
+ char *username;
+ char *schemaname;
+ char *tool_name;
+ char *match_pattern;
+ bool negate_match_pattern;
+ int re_modifiers; // bitmask: 1=CASELESS
+ int flagIN;
+ int flagOUT;
+ char *replace_pattern;
+ int timeout_ms;
+ char *error_msg;
+ char *ok_msg;
+ bool log;
+ bool apply;
+ char *comment;
+ uint64_t hits; // in-memory only, not persisted to table
+ void* regex_engine; // compiled regex (RE2)
+
+ MCP_Query_Rule() : rule_id(0), active(false), username(NULL), schemaname(NULL),
+ tool_name(NULL), match_pattern(NULL), negate_match_pattern(false),
+ re_modifiers(1), flagIN(0), flagOUT(0), replace_pattern(NULL),
+ timeout_ms(0), error_msg(NULL), ok_msg(NULL), log(false), apply(true),
+ comment(NULL), hits(0), regex_engine(NULL) {}
+};
+
+/**
+ * @brief MCP query digest statistics
+ */
+struct MCP_Query_Digest_Stats {
+ std::string tool_name;
+ int run_id;
+ uint64_t digest;
+ std::string digest_text;
+ unsigned int count_star;
+ time_t first_seen;
+ time_t last_seen;
+ unsigned long long sum_time;
+ unsigned long long min_time;
+ unsigned long long max_time;
+
+ MCP_Query_Digest_Stats() : run_id(-1), digest(0), count_star(0),
+ first_seen(0), last_seen(0),
+ sum_time(0), min_time(0), max_time(0) {}
+
+ void add_timing(unsigned long long duration_us, time_t timestamp) {
+ count_star++;
+ sum_time += duration_us;
+ if (duration_us < min_time || min_time == 0) min_time = duration_us;
+ if (duration_us > max_time) max_time = duration_us;
+ if (first_seen == 0) first_seen = timestamp;
+ last_seen = timestamp;
+ }
+};
+
+/**
+ * @brief MCP query processor output
+ *
+ * This structure collects all possible actions from matching MCP query rules.
+ * A single rule can perform multiple actions simultaneously (rewrite + timeout + block).
+ * Actions are inferred from rule properties:
+ * - if error_msg != NULL → block
+ * - if replace_pattern != NULL → rewrite
+ * - if timeout_ms > 0 → timeout
+ * - if OK_msg != NULL → return OK message
+ *
+ * The calling code checks these fields and performs the appropriate actions.
+ */
+struct MCP_Query_Processor_Output {
+ std::string *new_query; // Rewritten query (caller must delete)
+ int timeout_ms; // Query timeout in milliseconds (-1 = not set)
+ char *error_msg; // Error message to return (NULL = not set)
+ char *OK_msg; // OK message to return (NULL = not set)
+ int log; // Whether to log this query (-1 = not set, 0 = no, 1 = yes)
+ int next_query_flagIN; // Flag for next query (-1 = not set)
+
+ void init() {
+ new_query = NULL;
+ timeout_ms = -1;
+ error_msg = NULL;
+ OK_msg = NULL;
+ log = -1;
+ next_query_flagIN = -1;
+ }
+
+ void destroy() {
+ if (new_query) {
+ delete new_query;
+ new_query = NULL;
+ }
+ if (error_msg) {
+ free(error_msg);
+ error_msg = NULL;
+ }
+ if (OK_msg) {
+ free(OK_msg);
+ OK_msg = NULL;
+ }
+ }
+
+ MCP_Query_Processor_Output() {
+ init();
+ }
+
+ ~MCP_Query_Processor_Output() {
+ destroy();
+ }
+};
+
+/**
+ * @brief Two-Phase Discovery Catalog Schema Manager
+ *
+ * This class manages a comprehensive SQLite catalog for database discovery with two layers:
+ * 1. Deterministic Layer: Static metadata harvested from MySQL INFORMATION_SCHEMA
+ * 2. LLM Agent Layer: Semantic interpretations generated by LLM agents
+ *
+ * Schema separates deterministic metadata (runs, objects, columns, indexes, fks)
+ * from LLM-generated semantics (summaries, domains, metrics, question templates).
+ */
+class Discovery_Schema {
+private:
+ SQLite3DB* db;
+ std::string db_path;
+
+ // MCP query rules management
+ std::vector mcp_query_rules;
+ pthread_rwlock_t mcp_rules_lock;
+ volatile unsigned int mcp_rules_version;
+
+ // MCP query digest statistics
+ std::unordered_map> mcp_digest_umap;
+ pthread_rwlock_t mcp_digest_rwlock;
+
+ /**
+ * @brief Initialize catalog schema with all tables
+ * @return 0 on success, -1 on error
+ */
+ int init_schema();
+
+ /**
+ * @brief Create deterministic layer tables
+ * @return 0 on success, -1 on error
+ */
+ int create_deterministic_tables();
+
+ /**
+ * @brief Create LLM agent layer tables
+ * @return 0 on success, -1 on error
+ */
+ int create_llm_tables();
+
+ /**
+ * @brief Create FTS5 indexes
+ * @return 0 on success, -1 on error
+ */
+ int create_fts_tables();
+
+public:
+ /**
+ * @brief Constructor
+ * @param path Path to the catalog database file
+ */
+ Discovery_Schema(const std::string& path);
+
+ /**
+ * @brief Destructor
+ */
+ ~Discovery_Schema();
+
+ /**
+ * @brief Initialize the catalog database
+ * @return 0 on success, -1 on error
+ */
+ int init();
+
+ /**
+ * @brief Close the catalog database
+ */
+ void close();
+
+ /**
+ * @brief Resolve schema name or run_id to a run_id
+ *
+ * If input is a numeric run_id, returns it as-is.
+ * If input is a schema name, finds the latest run_id for that schema.
+ *
+ * @param run_id_or_schema Either a numeric run_id or a schema name
+ * @return run_id on success, -1 if schema not found
+ */
+ int resolve_run_id(const std::string& run_id_or_schema);
+
+ /**
+ * @brief Create a new discovery run
+ *
+ * @param source_dsn Data source identifier (e.g., "mysql://host:port/")
+ * @param mysql_version MySQL server version
+ * @param notes Optional notes for this run
+ * @return run_id on success, -1 on error
+ */
+ int create_run(
+ const std::string& source_dsn,
+ const std::string& mysql_version,
+ const std::string& notes = ""
+ );
+
+ /**
+ * @brief Finish a discovery run
+ *
+ * @param run_id The run ID to finish
+ * @param notes Optional completion notes
+ * @return 0 on success, -1 on error
+ */
+ int finish_run(int run_id, const std::string& notes = "");
+
+ /**
+ * @brief Get run ID info
+ *
+ * @param run_id The run ID
+ * @return JSON string with run info
+ */
+ std::string get_run_info(int run_id);
+
+ /**
+ * @brief Create a new LLM agent run bound to a deterministic run
+ *
+ * @param run_id The deterministic run ID
+ * @param model_name Model name (e.g., "claude-3.5-sonnet")
+ * @param prompt_hash Optional hash of system prompt
+ * @param budget_json Optional budget JSON
+ * @return agent_run_id on success, -1 on error
+ */
+ int create_agent_run(
+ int run_id,
+ const std::string& model_name,
+ const std::string& prompt_hash = "",
+ const std::string& budget_json = ""
+ );
+
+ /**
+ * @brief Finish an agent run
+ *
+ * @param agent_run_id The agent run ID
+ * @param status Status: "success" or "failed"
+ * @param error Optional error message
+ * @return 0 on success, -1 on error
+ */
+ int finish_agent_run(
+ int agent_run_id,
+ const std::string& status,
+ const std::string& error = ""
+ );
+
+ /**
+ * @brief Get the last (most recent) agent_run_id for a given run_id
+ *
+ * @param run_id Run ID
+ * @return agent_run_id on success, 0 if no agent runs exist for this run_id
+ */
+ int get_last_agent_run_id(int run_id);
+
+ /**
+ * @brief Insert a schema
+ *
+ * @param run_id Run ID
+ * @param schema_name Schema/database name
+ * @param charset Character set
+ * @param collation Collation
+ * @return schema_id on success, -1 on error
+ */
+ int insert_schema(
+ int run_id,
+ const std::string& schema_name,
+ const std::string& charset = "",
+ const std::string& collation = ""
+ );
+
+ /**
+ * @brief Insert an object (table/view/routine/trigger)
+ *
+ * @param run_id Run ID
+ * @param schema_name Schema name
+ * @param object_name Object name
+ * @param object_type Object type (table/view/routine/trigger)
+ * @param engine Storage engine (for tables)
+ * @param table_rows_est Estimated row count
+ * @param data_length Data length in bytes
+ * @param index_length Index length in bytes
+ * @param create_time Creation time
+ * @param update_time Last update time
+ * @param object_comment Object comment
+ * @param definition_sql Definition SQL (for views/routines)
+ * @return object_id on success, -1 on error
+ */
+ int insert_object(
+ int run_id,
+ const std::string& schema_name,
+ const std::string& object_name,
+ const std::string& object_type,
+ const std::string& engine = "",
+ long table_rows_est = 0,
+ long data_length = 0,
+ long index_length = 0,
+ const std::string& create_time = "",
+ const std::string& update_time = "",
+ const std::string& object_comment = "",
+ const std::string& definition_sql = ""
+ );
+
+ /**
+ * @brief Insert a column
+ *
+ * @param object_id Object ID
+ * @param ordinal_pos Ordinal position
+ * @param column_name Column name
+ * @param data_type Data type
+ * @param column_type Full column type
+ * @param is_nullable Is nullable (0/1)
+ * @param column_default Default value
+ * @param extra Extra info (auto_increment, etc.)
+ * @param charset Character set
+ * @param collation Collation
+ * @param column_comment Column comment
+ * @param is_pk Is primary key (0/1)
+ * @param is_unique Is unique (0/1)
+ * @param is_indexed Is indexed (0/1)
+ * @param is_time Is time type (0/1)
+ * @param is_id_like Is ID-like name (0/1)
+ * @return column_id on success, -1 on error
+ */
+ int insert_column(
+ int object_id,
+ int ordinal_pos,
+ const std::string& column_name,
+ const std::string& data_type,
+ const std::string& column_type = "",
+ int is_nullable = 1,
+ const std::string& column_default = "",
+ const std::string& extra = "",
+ const std::string& charset = "",
+ const std::string& collation = "",
+ const std::string& column_comment = "",
+ int is_pk = 0,
+ int is_unique = 0,
+ int is_indexed = 0,
+ int is_time = 0,
+ int is_id_like = 0
+ );
+
+ /**
+ * @brief Insert an index
+ *
+ * @param object_id Object ID
+ * @param index_name Index name
+ * @param is_unique Is unique (0/1)
+ * @param is_primary Is primary key (0/1)
+ * @param index_type Index type (BTREE/HASH/FULLTEXT)
+ * @param cardinality Cardinality
+ * @return index_id on success, -1 on error
+ */
+ int insert_index(
+ int object_id,
+ const std::string& index_name,
+ int is_unique = 0,
+ int is_primary = 0,
+ const std::string& index_type = "",
+ long cardinality = 0
+ );
+
+ /**
+ * @brief Insert an index column
+ *
+ * @param index_id Index ID
+ * @param seq_in_index Sequence in index
+ * @param column_name Column name
+ * @param sub_part Sub-part length
+ * @param collation Collation (A/D)
+ * @return 0 on success, -1 on error
+ */
+ int insert_index_column(
+ int index_id,
+ int seq_in_index,
+ const std::string& column_name,
+ int sub_part = 0,
+ const std::string& collation = "A"
+ );
+
+ /**
+ * @brief Insert a foreign key
+ *
+ * @param run_id Run ID
+ * @param child_object_id Child object ID
+ * @param fk_name FK name
+ * @param parent_schema_name Parent schema name
+ * @param parent_object_name Parent object name
+ * @param on_update ON UPDATE rule
+ * @param on_delete ON DELETE rule
+ * @return fk_id on success, -1 on error
+ */
+ int insert_foreign_key(
+ int run_id,
+ int child_object_id,
+ const std::string& fk_name,
+ const std::string& parent_schema_name,
+ const std::string& parent_object_name,
+ const std::string& on_update = "",
+ const std::string& on_delete = ""
+ );
+
+ /**
+ * @brief Insert a foreign key column
+ *
+ * @param fk_id FK ID
+ * @param seq Sequence number
+ * @param child_column Child column name
+ * @param parent_column Parent column name
+ * @return 0 on success, -1 on error
+ */
+ int insert_foreign_key_column(
+ int fk_id,
+ int seq,
+ const std::string& child_column,
+ const std::string& parent_column
+ );
+
+ /**
+ * @brief Update object derived flags
+ *
+ * Updates has_primary_key, has_foreign_keys, has_time_column flags
+ * based on actual data in columns, indexes, foreign_keys tables.
+ *
+ * @param run_id Run ID
+ * @return 0 on success, -1 on error
+ */
+ int update_object_flags(int run_id);
+
+ /**
+ * @brief Insert or update a profile
+ *
+ * @param run_id Run ID
+ * @param object_id Object ID
+ * @param profile_kind Profile kind (table_quick, column, time_range, etc.)
+ * @param profile_json Profile data as JSON string
+ * @return 0 on success, -1 on error
+ */
+ int upsert_profile(
+ int run_id,
+ int object_id,
+ const std::string& profile_kind,
+ const std::string& profile_json
+ );
+
+ /**
+ * @brief Rebuild FTS index for a run
+ *
+ * Deletes and rebuilds the fts_objects index for all objects in a run.
+ *
+ * @param run_id Run ID
+ * @return 0 on success, -1 on error
+ */
+ int rebuild_fts_index(int run_id);
+
+ /**
+ * @brief Full-text search over objects
+ *
+ * @param run_id Run ID
+ * @param query FTS5 query
+ * @param limit Max results
+ * @param object_type Optional filter by object type
+ * @param schema_name Optional filter by schema name
+ * @return JSON array of matching objects
+ */
+ std::string fts_search(
+ int run_id,
+ const std::string& query,
+ int limit = 25,
+ const std::string& object_type = "",
+ const std::string& schema_name = ""
+ );
+
+ /**
+ * @brief Get object by ID or key
+ *
+ * @param run_id Run ID
+ * @param object_id Object ID (optional)
+ * @param schema_name Schema name (if using object_key)
+ * @param object_name Object name (if using object_key)
+ * @param include_definition Include view/routine definitions
+ * @param include_profiles Include profile data
+ * @return JSON string with object details
+ */
+ std::string get_object(
+ int run_id,
+ int object_id = -1,
+ const std::string& schema_name = "",
+ const std::string& object_name = "",
+ bool include_definition = false,
+ bool include_profiles = true
+ );
+
+ /**
+ * @brief List objects with pagination
+ *
+ * @param run_id Run ID
+ * @param schema_name Optional schema filter
+ * @param object_type Optional object type filter
+ * @param order_by Order by field (name/rows_est_desc/size_desc)
+ * @param page_size Page size
+ * @param page_token Page token (empty for first page)
+ * @return JSON string with results and next page token
+ */
+ std::string list_objects(
+ int run_id,
+ const std::string& schema_name = "",
+ const std::string& object_type = "",
+ const std::string& order_by = "name",
+ int page_size = 50,
+ const std::string& page_token = ""
+ );
+
+ /**
+ * @brief Get relationships for an object
+ *
+ * Returns foreign keys, view dependencies, and inferred relationships.
+ *
+ * @param run_id Run ID
+ * @param object_id Object ID
+ * @param include_inferred Include LLM-inferred relationships
+ * @param min_confidence Minimum confidence for inferred relationships
+ * @return JSON string with relationships
+ */
+ std::string get_relationships(
+ int run_id,
+ int object_id,
+ bool include_inferred = true,
+ double min_confidence = 0.0
+ );
+
+ /**
+ * @brief Append an agent event
+ *
+ * @param agent_run_id Agent run ID
+ * @param event_type Event type (tool_call/tool_result/note/decision)
+ * @param payload_json Event payload as JSON string
+ * @return event_id on success, -1 on error
+ */
+ int append_agent_event(
+ int agent_run_id,
+ const std::string& event_type,
+ const std::string& payload_json
+ );
+
+ /**
+ * @brief Upsert an LLM object summary
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param object_id Object ID
+ * @param summary_json Summary data as JSON string
+ * @param confidence Confidence score (0.0-1.0)
+ * @param status Status (draft/validated/stable)
+ * @param sources_json Optional sources evidence
+ * @return 0 on success, -1 on error
+ */
+ int upsert_llm_summary(
+ int agent_run_id,
+ int run_id,
+ int object_id,
+ const std::string& summary_json,
+ double confidence = 0.5,
+ const std::string& status = "draft",
+ const std::string& sources_json = ""
+ );
+
+ /**
+ * @brief Get LLM summary for an object
+ *
+ * @param run_id Run ID
+ * @param object_id Object ID
+ * @param agent_run_id Optional specific agent run ID
+ * @param latest Get latest summary across all agent runs
+ * @return JSON string with summary or null
+ */
+ std::string get_llm_summary(
+ int run_id,
+ int object_id,
+ int agent_run_id = -1,
+ bool latest = true
+ );
+
+ /**
+ * @brief Upsert an LLM-inferred relationship
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param child_object_id Child object ID
+ * @param child_column Child column name
+ * @param parent_object_id Parent object ID
+ * @param parent_column Parent column name
+ * @param rel_type Relationship type (fk_like/bridge/polymorphic/etc)
+ * @param confidence Confidence score
+ * @param evidence_json Evidence JSON string
+ * @return 0 on success, -1 on error
+ */
+ int upsert_llm_relationship(
+ int agent_run_id,
+ int run_id,
+ int child_object_id,
+ const std::string& child_column,
+ int parent_object_id,
+ const std::string& parent_column,
+ const std::string& rel_type = "fk_like",
+ double confidence = 0.6,
+ const std::string& evidence_json = ""
+ );
+
+ /**
+ * @brief Upsert a domain
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param domain_key Domain key (e.g., "billing", "sales")
+ * @param title Domain title
+ * @param description Domain description
+ * @param confidence Confidence score
+ * @return domain_id on success, -1 on error
+ */
+ int upsert_llm_domain(
+ int agent_run_id,
+ int run_id,
+ const std::string& domain_key,
+ const std::string& title = "",
+ const std::string& description = "",
+ double confidence = 0.6
+ );
+
+ /**
+ * @brief Set domain members
+ *
+ * Replaces all members of a domain with the provided list.
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param domain_key Domain key
+ * @param members_json Members JSON array with object_id, role, confidence
+ * @return 0 on success, -1 on error
+ */
+ int set_domain_members(
+ int agent_run_id,
+ int run_id,
+ const std::string& domain_key,
+ const std::string& members_json
+ );
+
+ /**
+ * @brief Upsert a metric
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param metric_key Metric key (e.g., "orders.count")
+ * @param title Metric title
+ * @param description Metric description
+ * @param domain_key Optional domain key
+ * @param grain Grain (day/order/customer/etc)
+ * @param unit Unit (USD/count/ms/etc)
+ * @param sql_template Optional SQL template
+ * @param depends_json Optional dependencies JSON
+ * @param confidence Confidence score
+ * @return metric_id on success, -1 on error
+ */
+ int upsert_llm_metric(
+ int agent_run_id,
+ int run_id,
+ const std::string& metric_key,
+ const std::string& title,
+ const std::string& description = "",
+ const std::string& domain_key = "",
+ const std::string& grain = "",
+ const std::string& unit = "",
+ const std::string& sql_template = "",
+ const std::string& depends_json = "",
+ double confidence = 0.6
+ );
+
+ /**
+ * @brief Add a question template
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param title Template title
+ * @param question_nl Natural language question
+ * @param template_json Query plan template JSON
+ * @param example_sql Optional example SQL
+ * @param related_objects JSON array of related object names (tables/views)
+ * @param confidence Confidence score
+ * @return template_id on success, -1 on error
+ */
+ int add_question_template(
+ int agent_run_id,
+ int run_id,
+ const std::string& title,
+ const std::string& question_nl,
+ const std::string& template_json,
+ const std::string& example_sql = "",
+ const std::string& related_objects = "",
+ double confidence = 0.6
+ );
+
+ /**
+ * @brief Add an LLM note
+ *
+ * @param agent_run_id Agent run ID
+ * @param run_id Deterministic run ID
+ * @param scope Note scope (global/schema/object/domain)
+ * @param object_id Optional object ID
+ * @param domain_key Optional domain key
+ * @param title Note title
+ * @param body Note body
+ * @param tags_json Optional tags JSON array
+ * @return note_id on success, -1 on error
+ */
+ int add_llm_note(
+ int agent_run_id,
+ int run_id,
+ const std::string& scope,
+ int object_id = -1,
+ const std::string& domain_key = "",
+ const std::string& title = "",
+ const std::string& body = "",
+ const std::string& tags_json = ""
+ );
+
+ /**
+ * @brief Full-text search over LLM artifacts
+ *
+ * @param run_id Run ID
+ * @param query FTS query (empty to list all)
+ * @param limit Max results
+ * @param include_objects Include full object details for question templates
+ * @return JSON array of matching LLM artifacts with example_sql and related_objects
+ */
+ std::string fts_search_llm(
+ int run_id,
+ const std::string& query,
+ int limit = 25,
+ bool include_objects = false
+ );
+
+ /**
+ * @brief Log an LLM search query
+ *
+ * @param run_id Run ID
+ * @param query Search query string
+ * @param lmt Result limit
+ * @return 0 on success, -1 on error
+ */
+ int log_llm_search(
+ int run_id,
+ const std::string& query,
+ int lmt = 25
+ );
+
+ /**
+ * @brief Log MCP tool invocation via /mcp/query/ endpoint
+ * @param tool_name Name of the tool that was called
+ * @param schema Schema name (empty if not applicable)
+ * @param run_id Run ID (0 or -1 if not applicable)
+ * @param start_time Start monotonic time (microseconds)
+ * @param execution_time Execution duration (microseconds)
+ * @param error Error message (empty if success)
+ * @return 0 on success, -1 on error
+ */
+ int log_query_tool_call(
+ const std::string& tool_name,
+ const std::string& schema,
+ int run_id,
+ unsigned long long start_time,
+ unsigned long long execution_time,
+ const std::string& error
+ );
+
+ /**
+ * @brief Get database handle for direct access
+ * @return SQLite3DB pointer
+ */
+ SQLite3DB* get_db() { return db; }
+
+ /**
+ * @brief Get the database file path
+ * @return Database file path
+ */
+ std::string get_db_path() const { return db_path; }
+
+ // ============================================================
+ // MCP QUERY RULES
+ // ============================================================
+
+ /**
+ * @brief Load MCP query rules from SQLite
+ */
+ void load_mcp_query_rules(SQLite3_result* resultset);
+
+ /**
+ * @brief Evaluate MCP query rules for a tool invocation
+ * @return MCP_Query_Processor_Output object populated with actions from matching rules
+ * Caller is responsible for destroying the returned object.
+ */
+ MCP_Query_Processor_Output* evaluate_mcp_query_rules(
+ const std::string& tool_name,
+ const std::string& schemaname,
+ const nlohmann::json& arguments,
+ const std::string& original_query
+ );
+
+ /**
+ * @brief Get current MCP query rules as resultset
+ */
+ SQLite3_result* get_mcp_query_rules();
+
+ /**
+ * @brief Get stats for MCP query rules (hits per rule)
+ */
+ SQLite3_result* get_stats_mcp_query_rules();
+
+ // ============================================================
+ // MCP QUERY DIGEST
+ // ============================================================
+
+ /**
+ * @brief Update MCP query digest statistics
+ */
+ void update_mcp_query_digest(
+ const std::string& tool_name,
+ int run_id,
+ uint64_t digest,
+ const std::string& digest_text,
+ unsigned long long duration_us,
+ time_t timestamp
+ );
+
+ /**
+ * @brief Get MCP query digest statistics
+ * @param reset If true, reset stats after retrieval
+ */
+ SQLite3_result* get_mcp_query_digest(bool reset = false);
+
+ /**
+ * @brief Compute MCP query digest hash using SpookyHash
+ */
+ static uint64_t compute_mcp_digest(
+ const std::string& tool_name,
+ const nlohmann::json& arguments
+ );
+
+ /**
+ * @brief Fingerprint MCP query arguments (replace literals with ?)
+ */
+ static std::string fingerprint_mcp_args(const nlohmann::json& arguments);
+};
+
+#endif /* CLASS_DISCOVERY_SCHEMA_H */
diff --git a/include/GenAI_Thread.h b/include/GenAI_Thread.h
index ce4183ed36..6dfdf70397 100644
--- a/include/GenAI_Thread.h
+++ b/include/GenAI_Thread.h
@@ -230,6 +230,14 @@ class GenAI_Threads_Handler
// Vector storage configuration
char* genai_vector_db_path; ///< Vector database file path (default: /var/lib/proxysql/ai_features.db)
int genai_vector_dimension; ///< Embedding dimension (default: 1536)
+
+ // RAG configuration
+ bool genai_rag_enabled; ///< Enable RAG features (default: false)
+ int genai_rag_k_max; ///< Maximum k for search results (default: 50)
+ int genai_rag_candidates_max; ///< Maximum candidates for hybrid search (default: 500)
+ int genai_rag_query_max_bytes; ///< Maximum query length in bytes (default: 8192)
+ int genai_rag_response_max_bytes; ///< Maximum response size in bytes (default: 5000000)
+ int genai_rag_timeout_ms; ///< RAG operation timeout in ms (default: 2000)
} variables;
struct {
diff --git a/include/MCP_Thread.h b/include/MCP_Thread.h
index dca5900406..b87d74f706 100644
--- a/include/MCP_Thread.h
+++ b/include/MCP_Thread.h
@@ -17,6 +17,7 @@ class Admin_Tool_Handler;
class Cache_Tool_Handler;
class Observe_Tool_Handler;
class AI_Tool_Handler;
+class RAG_Tool_Handler;
/**
* @brief MCP Threads Handler class for managing MCP module configuration
@@ -56,8 +57,7 @@ class MCP_Threads_Handler
char* mcp_mysql_user; ///< MySQL username for tool connections
char* mcp_mysql_password; ///< MySQL password for tool connections
char* mcp_mysql_schema; ///< Default schema/database
- char* mcp_catalog_path; ///< Path to catalog SQLite database
- char* mcp_fts_path; ///< Path to FTS SQLite database
+ // Catalog path is hardcoded to mcp_catalog.db in the datadir
} variables;
/**
@@ -91,12 +91,14 @@ class MCP_Threads_Handler
/**
* @brief Pointers to the new dedicated tool handlers for each endpoint
*
- * Each endpoint now has its own dedicated tool handler:
+ * Each endpoint has its own dedicated tool handler:
* - config_tool_handler: /mcp/config endpoint
- * - query_tool_handler: /mcp/query endpoint
+ * - query_tool_handler: /mcp/query endpoint (includes two-phase discovery tools)
* - admin_tool_handler: /mcp/admin endpoint
* - cache_tool_handler: /mcp/cache endpoint
* - observe_tool_handler: /mcp/observe endpoint
+ * - ai_tool_handler: /mcp/ai endpoint
+ * - rag_tool_handler: /mcp/rag endpoint
*/
Config_Tool_Handler* config_tool_handler;
Query_Tool_Handler* query_tool_handler;
@@ -104,6 +106,7 @@ class MCP_Threads_Handler
Cache_Tool_Handler* cache_tool_handler;
Observe_Tool_Handler* observe_tool_handler;
AI_Tool_Handler* ai_tool_handler;
+ RAG_Tool_Handler* rag_tool_handler;
/**
diff --git a/include/MySQL_Catalog.h b/include/MySQL_Catalog.h
index 233895c010..b57df1422f 100644
--- a/include/MySQL_Catalog.h
+++ b/include/MySQL_Catalog.h
@@ -60,14 +60,16 @@ class MySQL_Catalog {
/**
* @brief Catalog upsert - create or update a catalog entry
*
+ * @param schema Schema name (e.g., "sales", "production") - empty for all schemas
* @param kind The kind of entry ("table", "view", "domain", "metric", "note")
- * @param key Unique key (e.g., "db.sales.orders")
+ * @param key Unique key (e.g., "orders", "customer_summary")
* @param document JSON document with summary/details
* @param tags Optional comma-separated tags
* @param links Optional comma-separated links to related keys
* @return 0 on success, -1 on error
*/
int upsert(
+ const std::string& schema,
const std::string& kind,
const std::string& key,
const std::string& document,
@@ -76,14 +78,16 @@ class MySQL_Catalog {
);
/**
- * @brief Get a catalog entry by kind and key
+ * @brief Get a catalog entry by schema, kind and key
*
+ * @param schema Schema name (empty for all schemas)
* @param kind The kind of entry
* @param key The unique key
* @param document Output: JSON document
* @return 0 on success, -1 if not found
*/
int get(
+ const std::string& schema,
const std::string& kind,
const std::string& key,
std::string& document
@@ -92,6 +96,7 @@ class MySQL_Catalog {
/**
* @brief Search catalog entries
*
+ * @param schema Schema name to filter (empty for all schemas)
* @param query Search query (searches in key, document, tags)
* @param kind Optional filter by kind
* @param tags Optional filter by tags (comma-separated)
@@ -100,6 +105,7 @@ class MySQL_Catalog {
* @return JSON array of matching entries
*/
std::string search(
+ const std::string& schema,
const std::string& query,
const std::string& kind = "",
const std::string& tags = "",
@@ -110,12 +116,14 @@ class MySQL_Catalog {
/**
* @brief List catalog entries with pagination
*
+ * @param schema Schema name to filter (empty for all schemas)
* @param kind Optional filter by kind
* @param limit Max results per page (default 50)
* @param offset Pagination offset (default 0)
* @return JSON array of entries with total count
*/
std::string list(
+ const std::string& schema = "",
const std::string& kind = "",
int limit = 50,
int offset = 0
@@ -140,11 +148,13 @@ class MySQL_Catalog {
/**
* @brief Delete a catalog entry
*
+ * @param schema Schema name (empty for all schemas)
* @param kind The kind of entry
* @param key The unique key
* @return 0 on success, -1 if not found
*/
int remove(
+ const std::string& schema,
const std::string& kind,
const std::string& key
);
diff --git a/include/MySQL_Tool_Handler.h b/include/MySQL_Tool_Handler.h
index bb2e010f9f..459c0077d7 100644
--- a/include/MySQL_Tool_Handler.h
+++ b/include/MySQL_Tool_Handler.h
@@ -331,11 +331,13 @@ class MySQL_Tool_Handler {
* @param kind Entry kind
* @param key Unique key
* @param document JSON document
+ * @param schema Schema name (empty for all schemas)
* @param tags Comma-separated tags
* @param links Comma-separated links
* @return JSON result
*/
std::string catalog_upsert(
+ const std::string& schema,
const std::string& kind,
const std::string& key,
const std::string& document,
@@ -345,14 +347,16 @@ class MySQL_Tool_Handler {
/**
* @brief Get catalog entry
+ * @param schema Schema name (empty for all schemas)
* @param kind Entry kind
* @param key Unique key
* @return JSON document or error
*/
- std::string catalog_get(const std::string& kind, const std::string& key);
+ std::string catalog_get(const std::string& schema, const std::string& kind, const std::string& key);
/**
* @brief Search catalog
+ * @param schema Schema name (empty for all schemas)
* @param query Search query
* @param kind Optional kind filter
* @param tags Optional tag filter
@@ -361,6 +365,7 @@ class MySQL_Tool_Handler {
* @return JSON array of matching entries
*/
std::string catalog_search(
+ const std::string& schema,
const std::string& query,
const std::string& kind = "",
const std::string& tags = "",
@@ -370,12 +375,14 @@ class MySQL_Tool_Handler {
/**
* @brief List catalog entries
+ * @param schema Schema name (empty for all schemas)
* @param kind Optional kind filter
* @param limit Max results per page (default 50)
* @param offset Pagination offset (default 0)
* @return JSON with total count and results array
*/
std::string catalog_list(
+ const std::string& schema = "",
const std::string& kind = "",
int limit = 50,
int offset = 0
@@ -398,11 +405,12 @@ class MySQL_Tool_Handler {
/**
* @brief Delete catalog entry
+ * @param schema Schema name (empty for all schemas)
* @param kind Entry kind
* @param key Unique key
* @return JSON result
*/
- std::string catalog_delete(const std::string& kind, const std::string& key);
+ std::string catalog_delete(const std::string& schema, const std::string& kind, const std::string& key);
// ========== FTS Tools (Full Text Search) ==========
diff --git a/include/ProxySQL_Admin_Tables_Definitions.h b/include/ProxySQL_Admin_Tables_Definitions.h
index 392df01745..451e4b614b 100644
--- a/include/ProxySQL_Admin_Tables_Definitions.h
+++ b/include/ProxySQL_Admin_Tables_Definitions.h
@@ -322,6 +322,98 @@
#define STATS_SQLITE_TABLE_PGSQL_QUERY_DIGEST_RESET "CREATE TABLE stats_pgsql_query_digest_reset (hostgroup INT , database VARCHAR NOT NULL , username VARCHAR NOT NULL , client_address VARCHAR NOT NULL , digest VARCHAR NOT NULL , digest_text VARCHAR NOT NULL , count_star INTEGER NOT NULL , first_seen INTEGER NOT NULL , last_seen INTEGER NOT NULL , sum_time INTEGER NOT NULL , min_time INTEGER NOT NULL , max_time INTEGER NOT NULL , sum_rows_affected INTEGER NOT NULL , sum_rows_sent INTEGER NOT NULL , PRIMARY KEY(hostgroup, database, username, client_address, digest))"
#define STATS_SQLITE_TABLE_PGSQL_PREPARED_STATEMENTS_INFO "CREATE TABLE stats_pgsql_prepared_statements_info (global_stmt_id INT NOT NULL , database VARCHAR NOT NULL , username VARCHAR NOT NULL , digest VARCHAR NOT NULL , ref_count_client INT NOT NULL , ref_count_server INT NOT NULL , num_param_types INT NOT NULL , query VARCHAR NOT NULL)"
+#define STATS_SQLITE_TABLE_MCP_QUERY_TOOLS_COUNTERS "CREATE TABLE stats_mcp_query_tools_counters (tool VARCHAR NOT NULL , schema VARCHAR NOT NULL , count INT NOT NULL , first_seen INTEGER NOT NULL , last_seen INTEGER NOT NULL , sum_time INTEGER NOT NULL , min_time INTEGER NOT NULL , max_time INTEGER NOT NULL , PRIMARY KEY (tool, schema))"
+#define STATS_SQLITE_TABLE_MCP_QUERY_TOOLS_COUNTERS_RESET "CREATE TABLE stats_mcp_query_tools_counters_reset (tool VARCHAR NOT NULL , schema VARCHAR NOT NULL , count INT NOT NULL , first_seen INTEGER NOT NULL , last_seen INTEGER NOT NULL , sum_time INTEGER NOT NULL , min_time INTEGER NOT NULL , max_time INTEGER NOT NULL , PRIMARY KEY (tool, schema))"
+
+// MCP query rules table - for firewall and query rewriting
+// Action is inferred from rule properties:
+// - if error_msg is not NULL → block
+// - if replace_pattern is not NULL → rewrite
+// - if timeout_ms > 0 → timeout
+// - otherwise → allow
+#define ADMIN_SQLITE_TABLE_MCP_QUERY_RULES "CREATE TABLE mcp_query_rules (" \
+ " rule_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL ," \
+ " active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 0 ," \
+ " username VARCHAR ," \
+ " schemaname VARCHAR ," \
+ " tool_name VARCHAR ," \
+ " match_pattern VARCHAR ," \
+ " negate_match_pattern INT CHECK (negate_match_pattern IN (0,1)) NOT NULL DEFAULT 0 ," \
+ " re_modifiers VARCHAR DEFAULT 'CASELESS' ," \
+ " flagIN INT NOT NULL DEFAULT 0 ," \
+ " flagOUT INT CHECK (flagOUT >= 0) ," \
+ " replace_pattern VARCHAR ," \
+ " timeout_ms INT CHECK (timeout_ms >= 0) ," \
+ " error_msg VARCHAR ," \
+ " OK_msg VARCHAR ," \
+ " log INT CHECK (log IN (0,1)) ," \
+ " apply INT CHECK (apply IN (0,1)) NOT NULL DEFAULT 1 ," \
+ " comment VARCHAR" \
+ ")"
+
+// MCP query rules runtime table - shows in-memory state of active rules
+// This table has the same schema as mcp_query_rules (no hits column).
+// The hits counter is only available in stats_mcp_query_rules table.
+// When this table is queried, it is automatically refreshed from the in-memory rules.
+#define ADMIN_SQLITE_TABLE_RUNTIME_MCP_QUERY_RULES "CREATE TABLE runtime_mcp_query_rules (" \
+ " rule_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL ," \
+ " active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 0 ," \
+ " username VARCHAR ," \
+ " schemaname VARCHAR ," \
+ " tool_name VARCHAR ," \
+ " match_pattern VARCHAR ," \
+ " negate_match_pattern INT CHECK (negate_match_pattern IN (0,1)) NOT NULL DEFAULT 0 ," \
+ " re_modifiers VARCHAR DEFAULT 'CASELESS' ," \
+ " flagIN INT NOT NULL DEFAULT 0 ," \
+ " flagOUT INT CHECK (flagOUT >= 0) ," \
+ " replace_pattern VARCHAR ," \
+ " timeout_ms INT CHECK (timeout_ms >= 0) ," \
+ " error_msg VARCHAR ," \
+ " OK_msg VARCHAR ," \
+ " log INT CHECK (log IN (0,1)) ," \
+ " apply INT CHECK (apply IN (0,1)) NOT NULL DEFAULT 1 ," \
+ " comment VARCHAR" \
+ ")"
+
+// MCP query digest statistics table
+#define STATS_SQLITE_TABLE_MCP_QUERY_DIGEST "CREATE TABLE stats_mcp_query_digest (" \
+ " tool_name VARCHAR NOT NULL ," \
+ " run_id INT ," \
+ " digest VARCHAR NOT NULL ," \
+ " digest_text VARCHAR NOT NULL ," \
+ " count_star INTEGER NOT NULL ," \
+ " first_seen INTEGER NOT NULL ," \
+ " last_seen INTEGER NOT NULL ," \
+ " sum_time INTEGER NOT NULL ," \
+ " min_time INTEGER NOT NULL ," \
+ " max_time INTEGER NOT NULL ," \
+ " PRIMARY KEY(tool_name, run_id, digest)" \
+ ")"
+
+// MCP query digest reset table
+#define STATS_SQLITE_TABLE_MCP_QUERY_DIGEST_RESET "CREATE TABLE stats_mcp_query_digest_reset (" \
+ " tool_name VARCHAR NOT NULL ," \
+ " run_id INT ," \
+ " digest VARCHAR NOT NULL ," \
+ " digest_text VARCHAR NOT NULL ," \
+ " count_star INTEGER NOT NULL ," \
+ " first_seen INTEGER NOT NULL ," \
+ " last_seen INTEGER NOT NULL ," \
+ " sum_time INTEGER NOT NULL ," \
+ " min_time INTEGER NOT NULL ," \
+ " max_time INTEGER NOT NULL ," \
+ " PRIMARY KEY(tool_name, run_id, digest)" \
+ ")"
+
+// MCP query rules statistics table - shows hit counters for each rule
+// This table contains only rule_id and hits count.
+// It is automatically populated when stats_mcp_query_rules is queried.
+// The hits counter increments each time a rule matches during query processing.
+#define STATS_SQLITE_TABLE_MCP_QUERY_RULES "CREATE TABLE stats_mcp_query_rules (" \
+ " rule_id INTEGER PRIMARY KEY NOT NULL ," \
+ " hits INTEGER NOT NULL" \
+ ")"
+
//#define STATS_SQLITE_TABLE_MEMORY_METRICS "CREATE TABLE stats_memory_metrics (Variable_Name VARCHAR NOT NULL PRIMARY KEY , Variable_Value VARCHAR NOT NULL)"
diff --git a/include/Query_Tool_Handler.h b/include/Query_Tool_Handler.h
index da067a6863..0bf8d02209 100644
--- a/include/Query_Tool_Handler.h
+++ b/include/Query_Tool_Handler.h
@@ -2,47 +2,92 @@
#define CLASS_QUERY_TOOL_HANDLER_H
#include "MCP_Tool_Handler.h"
-#include "MySQL_Tool_Handler.h"
+#include "Discovery_Schema.h"
+#include "Static_Harvester.h"
#include
/**
* @brief Query Tool Handler for /mcp/query endpoint
*
* This handler provides tools for safe database exploration and query execution.
- * It wraps the existing MySQL_Tool_Handler to provide MCP protocol compliance.
+ * It now uses the comprehensive Discovery_Schema for catalog operations and includes
+ * the two-phase discovery tools.
*
* Tools provided:
- * - list_schemas: List databases
- * - list_tables: List tables in schema
- * - describe_table: Get table structure
- * - get_constraints: Get foreign keys and constraints
- * - table_profile: Get table statistics
- * - column_profile: Get column statistics
- * - sample_rows: Get sample data
- * - sample_distinct: Sample distinct values
- * - run_sql_readonly: Execute read-only SQL
- * - explain_sql: Explain query execution plan
- * - suggest_joins: Suggest table joins
- * - find_reference_candidates: Find foreign key references
- * - catalog_upsert: Store data in catalog
- * - catalog_get: Retrieve from catalog
- * - catalog_search: Search catalog
- * - catalog_list: List catalog entries
- * - catalog_merge: Merge catalog entries
- * - catalog_delete: Delete from catalog
+ * - Inventory: list_schemas, list_tables, describe_table, get_constraints
+ * - Profiling: table_profile, column_profile
+ * - Sampling: sample_rows, sample_distinct
+ * - Query: run_sql_readonly, explain_sql
+ * - Relationships: suggest_joins, find_reference_candidates
+ * - Discovery (NEW): discovery.run_static, agent.*, llm.*
+ * - Catalog (NEW): All catalog tools now use Discovery_Schema
*/
class Query_Tool_Handler : public MCP_Tool_Handler {
private:
- MySQL_Tool_Handler* mysql_handler; ///< Underlying MySQL tool handler
- bool owns_handler; ///< Whether we created the handler
+ // MySQL connection configuration
+ std::string mysql_hosts;
+ std::string mysql_ports;
+ std::string mysql_user;
+ std::string mysql_password;
+ std::string mysql_schema;
+
+ // Discovery components (NEW - replaces MySQL_Tool_Handler wrapper)
+ Discovery_Schema* catalog; ///< Discovery catalog (replaces old MySQL_Catalog)
+ Static_Harvester* harvester; ///< Static harvester for Phase 1
+
+ // Connection pool for MySQL queries
+ struct MySQLConnection {
+ void* mysql; ///< MySQL connection handle (MYSQL*)
+ std::string host;
+ int port;
+ bool in_use;
+ std::string current_schema; ///< Track current schema for this connection
+ };
+ std::vector connection_pool;
+ pthread_mutex_t pool_lock;
+ int pool_size;
+
+ // Query guardrails
+ int max_rows;
+ int timeout_ms;
+ bool allow_select_star;
+
+ // Statistics for a specific (tool, schema) pair
+ struct ToolUsageStats {
+ unsigned long long count;
+ unsigned long long first_seen;
+ unsigned long long last_seen;
+ unsigned long long sum_time;
+ unsigned long long min_time;
+ unsigned long long max_time;
+
+ ToolUsageStats() : count(0), first_seen(0), last_seen(0),
+ sum_time(0), min_time(0), max_time(0) {}
+
+ void add_timing(unsigned long long duration, unsigned long long timestamp) {
+ count++;
+ sum_time += duration;
+ if (duration < min_time || min_time == 0) {
+ if (duration) min_time = duration;
+ }
+ if (duration > max_time) {
+ max_time = duration;
+ }
+ if (first_seen == 0) {
+ first_seen = timestamp;
+ }
+ last_seen = timestamp;
+ }
+ };
+
+ // Tool usage counters: tool_name -> schema_name -> ToolUsageStats
+ typedef std::map SchemaStatsMap;
+ typedef std::map ToolUsageStatsMap;
+ ToolUsageStatsMap tool_usage_stats;
+ pthread_mutex_t counters_lock;
/**
* @brief Create tool list schema for a tool
- * @param tool_name Name of the tool
- * @param description Description of the tool
- * @param required_params Required parameter names
- * @param optional_params Optional parameter names with types
- * @return JSON schema object
*/
json create_tool_schema(
const std::string& tool_name,
@@ -51,21 +96,61 @@ class Query_Tool_Handler : public MCP_Tool_Handler {
const std::map& optional_params
);
-public:
/**
- * @brief Constructor with existing MySQL_Tool_Handler
- * @param handler Existing MySQL_Tool_Handler to wrap
+ * @brief Initialize MySQL connection pool
+ */
+ int init_connection_pool();
+
+ /**
+ * @brief Get a connection from the pool
*/
- Query_Tool_Handler(MySQL_Tool_Handler* handler);
+ void* get_connection();
/**
- * @brief Constructor creating new MySQL_Tool_Handler
- * @param hosts Comma-separated list of MySQL hosts
- * @param ports Comma-separated list of MySQL ports
- * @param user MySQL username
- * @param password MySQL password
- * @param schema Default schema/database
- * @param catalog_path Path to catalog database
+ * @brief Return a connection to the pool
+ */
+ void return_connection(void* mysql);
+
+ /**
+ * @brief Find connection wrapper by mysql pointer (for internal use)
+ * @param mysql_ptr MySQL connection pointer
+ * @return Pointer to connection wrapper, or nullptr if not found
+ * @note Caller should NOT hold pool_lock when calling this
+ */
+ MySQLConnection* find_connection(void* mysql_ptr);
+
+ /**
+ * @brief Execute a query and return results as JSON
+ */
+ std::string execute_query(const std::string& query);
+
+ /**
+ * @brief Execute a query with optional schema switching
+ * @param query SQL query to execute
+ * @param schema Schema name to switch to (empty = use default)
+ * @return JSON result with success flag and rows/error
+ */
+ std::string execute_query_with_schema(
+ const std::string& query,
+ const std::string& schema
+ );
+
+ /**
+ * @brief Validate SQL is read-only
+ */
+ bool validate_readonly_query(const std::string& query);
+
+ /**
+ * @brief Check if SQL contains dangerous keywords
+ */
+ bool is_dangerous_query(const std::string& query);
+
+ // Friend function for tracking tool invocations
+ friend void track_tool_invocation(Query_Tool_Handler*, const std::string&, const std::string&, unsigned long long);
+
+public:
+ /**
+ * @brief Constructor (creates catalog and harvester)
*/
Query_Tool_Handler(
const std::string& hosts,
@@ -90,10 +175,27 @@ class Query_Tool_Handler : public MCP_Tool_Handler {
std::string get_handler_name() const override { return "query"; }
/**
- * @brief Get the underlying MySQL_Tool_Handler
- * @return Pointer to MySQL_Tool_Handler
+ * @brief Get the discovery catalog
+ */
+ Discovery_Schema* get_catalog() const { return catalog; }
+
+ /**
+ * @brief Get the static harvester
+ */
+ Static_Harvester* get_harvester() const { return harvester; }
+
+ /**
+ * @brief Get tool usage statistics (thread-safe copy)
+ * @return ToolUsageStatsMap copy with tool_name -> schema_name -> ToolUsageStats
+ */
+ ToolUsageStatsMap get_tool_usage_stats();
+
+ /**
+ * @brief Get tool usage statistics as SQLite3_result* with optional reset
+ * @param reset If true, resets internal counters after capturing data
+ * @return SQLite3_result* with columns: tool, schema, count, first_seen, last_seen, sum_time, min_time, max_time. Caller must delete.
*/
- MySQL_Tool_Handler* get_mysql_handler() const { return mysql_handler; }
+ SQLite3_result* get_tool_usage_stats_resultset(bool reset = false);
};
#endif /* CLASS_QUERY_TOOL_HANDLER_H */
diff --git a/include/RAG_Tool_Handler.h b/include/RAG_Tool_Handler.h
new file mode 100644
index 0000000000..b4de86d3cf
--- /dev/null
+++ b/include/RAG_Tool_Handler.h
@@ -0,0 +1,451 @@
+/**
+ * @file RAG_Tool_Handler.h
+ * @brief RAG Tool Handler for MCP protocol
+ *
+ * Provides RAG (Retrieval-Augmented Generation) tools via MCP protocol including:
+ * - FTS search over documents
+ * - Vector search over embeddings
+ * - Hybrid search combining FTS and vectors
+ * - Fetch tools for retrieving document/chunk content
+ * - Refetch tool for authoritative source data
+ * - Admin tools for operational visibility
+ *
+ * The RAG subsystem implements a complete retrieval system with:
+ * - Full-text search using SQLite FTS5
+ * - Semantic search using vector embeddings with sqlite3-vec
+ * - Hybrid search combining both approaches
+ * - Comprehensive filtering capabilities
+ * - Security features including input validation and limits
+ * - Performance optimizations
+ *
+ * @date 2026-01-19
+ * @author ProxySQL Team
+ * @copyright GNU GPL v3
+ * @ingroup mcp
+ * @ingroup rag
+ */
+
+#ifndef CLASS_RAG_TOOL_HANDLER_H
+#define CLASS_RAG_TOOL_HANDLER_H
+
+#include "MCP_Tool_Handler.h"
+#include "sqlite3db.h"
+#include "GenAI_Thread.h"
+#include
+#include
+#include