Feature: RAG

.gitignore

@@ -183,6 +183,10 @@ dvclive
 # MacOS
 *.DS_Store
 
+.prettierrc
+
+*.ipynb
+*.pkl
 job_queue.db-shm
 job_queue.db-wal
 
@@ -202,3 +206,8 @@ docs/node_modules/
 !docs/docs/build/**
 !docs/i18n/**/build/
 !docs/i18n/**/build/**
+
+**/csv-train.arrow
+**/dataset_info.json
+
+/RAG_benchmark

AGENTS.md

@@ -0,0 +1,126 @@
+# AGENTS.md
+
+This file provides guidance to AI coding agents when working with code in this repository.
+
+## What is DashAI
+
+DashAI is a desktop/web graphical toolbox for training, evaluating, and deploying ML models. FastAPI backend + React frontend, optionally wrapped in PyWebView for a native desktop window. Python >= 3.10.
+
+## Commands
+
+### Backend
+
+```bash
+pip install -e . -r requirements-dev.txt
+pre-commit install
+
+# Run dev server
+python -m DashAI --no-browser --logging-level DEBUG
+
+# Lint / format (must pass both before committing)
+ruff check --fix
+ruff format
+
+# Run all tests (in-memory SQLite — no setup needed)
+pytest tests/
+
+# Run a single test
+pytest tests/back/api/test_components_api.py -v
+pytest tests/back/api/test_components_api.py::test_function_name -v
+
+# Database migrations (auto-runs on startup)
+alembic upgrade head
+```
+
+### Frontend
+
+```bash
+cd DashAI/front
+
+yarn install        # requires Node LTS + Yarn 3.5.0 (enforced by packageManager)
+yarn start          # dev server on http://localhost:3000 (eslint disabled)
+yarn build          # eslint disabled
+yarn test           # eslint disabled
+yarn test FileName.test.tsx
+yarn lint           # runs eslint explicitly (uses eslint.config.js)
+```
+
+**Important:** `start`, `build`, and `test` scripts all set `DISABLE_ESLINT_PLUGIN=true`. ESLint only runs via the dedicated `yarn lint` command. Prettier is configured in `DashAI/front/.prettierrc` and runs via pre-commit.
+
+## Architecture
+
+```
+Browser / PyWebView
+  → React (port 3000 dev / port 8000 prod)
+  → FastAPI (/api/v1/...)
+  → Service layer → SQLite (SQLAlchemy + Alembic)
+  → Huey job queue (async: training, conversion, exploration)
+```
+
+The frontend polls `/api/v1/jobs/{job_id}` for long-running task status.
+
+## Key files
+
+| Path | Purpose |
+|------|---------|
+| `DashAI/__main__.py` | CLI entry point (typer). Starts uvicorn + Huey consumer subprocess |
+| `DashAI/back/app.py` | FastAPI `create_app` factory |
+| `DashAI/back/container.py` | DI container (`kink`) — config, DB engine, ComponentRegistry, job queue |
+| `DashAI/back/initial_components.py` | **Registers all components on startup.** Add new ML components here in `get_initial_components()` |
+| `DashAI/back/dependencies/config_builder.py` | Builds config dict (paths, logging, calls `get_initial_components()`) |
+| `DashAI/back/dependencies/registry/component_registry.py` | `ComponentRegistry` — resolves components by name string |
+| `DashAI/back/api/api_v1/endpoints/` | REST endpoints — each file is a FastAPI router |
+| `DashAI/back/api/api_v1/api.py` | Mounts all endpoint routers on `api_router_v1` |
+| `DashAI/back/core/schema_fields/` | Type system driving dynamic frontend forms |
+| `DashAI/back/pipeline/` | DAG pipeline nodes |
+| `DashAI/back/plugins/` | Plugin system (PyPI packages with `dashai.plugins` entry point) |
+| `DashAI/front/src/components/configurableObject/` | Auto-generates forms from backend component schemas |
+| `DashAI/front/src/pages/generative/SessionRouter.jsx` | Routes `/app/generative/sessions/:id` to RAG or non-RAG view based on session `task_name` |
+| `DashAI/front/src/pages/generative/RAGSession/` | RAG session setup wizard (RAGSessionSetup) + collapsible section components per pipeline stage |
+| `DashAI/front/src/pages/generative/RAGSession/sections/` | Per-stage components: ChunkingSection, RetrieverSection, GeneratorSection, PromptSection |
+| `DashAI/front/src/pages/generative/RAGSession/advanced/` | Advanced configuration modals: CompositeRetrieverBuilder, ChunkingConfigurationStep, RetrieverConfigurationStep, etc. |
+| `DashAI/front/src/pages/generative/RAGSession/components/` | Reusable bodies: GeneratorBody, PromptBody, PresetCard, AdvancedConfigCard |
+
+## Key patterns
+
+**ComponentRegistry** — all ML components (models, metrics, converters, dataloaders, explorers, explainers, tasks, optimizers, jobs, pipeline nodes) are registered at startup and resolved by name string. To add a new component: subclass the relevant base, define its schema, and add it to `get_initial_components()` in `DashAI/back/initial_components.py`.
+
+**Schema / type system** — every component declares parameters using `BaseSchema` + field classes (`IntField`, `FloatField`, `ComponentField`, `UnionType`, etc.) with `MultilingualString` labels. The frontend uses these schemas to auto-generate configuration forms.
+
+**Dependency injection (`kink`)** — singletons (config, DB engine, ComponentRegistry, job queue) live in the `di` container. Use `@inject` to receive them. Config is accessed as `di["config"]` and includes `INITIAL_COMPONENTS`.
+
+**Huey job queue** — in dev mode the Huey consumer runs as a subprocess spawned from `__main__.py`; in PyInstaller bundles it runs as a daemon thread due to limitations with frozen executables.
+
+## RAG Module
+
+DashAI includes a **Retrieval-Augmented Generation (RAG)** module — a 4-stage pipeline (Document Loading → Chunking → Retrieval → Generation) for chatting with your documents.
+
+- **Backend:** `DashAI/back/models/RAG/` — pipeline orchestrator (RAGPipeline), abstract factory (RAGModelsFactory), sub-factories for prompts/chunking/retrievers/LLMs, retriever repository (all SQL), document loader, chunk-set caching.
+- **Frontend:** `DashAI/front/src/pages/generative/RAGSession/` — session setup wizard (RAGSessionSetup, RAGSessionPage) with stage sections (ChunkingSection, RetrieverSection, GeneratorSection, PromptSection) and advanced config modals.
+- **Jobs:** `RAGJob` extends `GenerativeJob` to run the RAG pipeline as a background task.
+- **Full docs (backend + frontend):** See [`docs/rag/`](./docs/rag/) for architecture, data models, retrievers, pipeline orchestration, frontend architecture, testing guide, and known constraints.
+
+## Testing
+
+- Backend tests use in-memory SQLite — no setup needed.
+- Key fixtures in `tests/back/conftest.py`: `test_path`, `test_datasets_path`, `test_job_queue`.
+- Job queue tests: `test_job_queue.set_test_mode(immediate=True)` runs tasks synchronously.
+- API tests use FastAPI `TestClient` defined in `tests/back/api/conftest.py` (module-scoped).
+- Frontend test command passes `--passWithNoTests` in CI.
+
+## Adding new things
+
+**New ML model or converter:** subclass `BaseModel` / `BaseConverter`, define schema, register in `DashAI/back/initial_components.py::get_initial_components()`.
+
+**New API endpoint:** add file in `DashAI/back/api/api_v1/endpoints/`, include its router in `DashAI/back/api/api_v1/api.py`.
+
+**New async job:** subclass `BaseJob`, dispatch via `job_queue.enqueue(...)`, track status through jobs API.
+
+## Data at runtime
+
+Default `~/.DashAI/` (overridable via `DASHAI_LOCAL_PATH` env var):
+`datasets/`, `runs/`, `explanations/`, `notebooks/`, `images/`, `documents/`, `rag/`, `sqlite.db`
+
+## CI
+
+Only a `publish.yml` workflow exists (on push to `production` branch). It builds the frontend, publishes to PyPI, and builds Windows/macOS installers. No CI test workflow is defined.

DashAI/__main__.py

@@ -65,7 +65,7 @@
 
 
 def open_browser() -> None:
-    _wait_for_backend_server(timeout=120)
+    _wait_for_backend_server(timeout=1200)
     url = "http://localhost:8000/app/"
     webbrowser.open(url=url, new=0, autoraise=True)
 
@@ -119,10 +119,11 @@ def _wait_for_backend_server(host="127.0.0.1", port=8000, timeout=15):
     import socket
     import time
 
+    timeout = 1000
     start_time = time.time()
     while time.time() - start_time < timeout:
         try:
-            with socket.create_connection((host, port), timeout=1):
+            with socket.create_connection((host, port), timeout=timeout):
                 return True
         except (OSError, ConnectionRefusedError):
             time.sleep(0.5)

DashAI/alembic/versions/8d5416cac8f1_merge_rag_and_production_heads.py

@@ -0,0 +1,26 @@
+"""Merge RAG and production heads
+
+Revision ID: 8d5416cac8f1
+Revises: d4e5f6a7b8c9, f1a2b3c4d5e6
+Create Date: 2026-07-06 12:30:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '8d5416cac8f1'
+down_revision: Union[str, None] = ('d4e5f6a7b8c9', 'f1a2b3c4d5e6')
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    pass
+
+
+def downgrade() -> None:
+    pass

DashAI/alembic/versions/a1b2c3d4e5f6_add_composite_retriever_support.py

@@ -0,0 +1,47 @@
+"""add composite retriever support
+
+Revision ID: a1b2c3d4e5f6
+Revises: f928e0b5203d
+Create Date: 2026-05-22 12:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'a1b2c3d4e5f6'
+down_revision: Union[str, None] = 'f928e0b5203d'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table('rag_retriever', schema=None) as batch_op:
+        batch_op.drop_constraint('chk_retriever_type', type_='check')
+        batch_op.add_column(
+            sa.Column('composite_child_ids', sa.JSON(), nullable=True)
+        )
+        batch_op.create_check_constraint(
+            'chk_retriever_type',
+            'NOT (dense_retriever_id IS NOT NULL AND sparse_retriever_id IS NOT NULL)'
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table('rag_retriever', schema=None) as batch_op:
+        batch_op.drop_constraint('chk_retriever_type', type_='check')
+        batch_op.drop_column('composite_child_ids')
+        batch_op.create_check_constraint(
+            'chk_retriever_type',
+            (
+                "(class_name = 'SparseRetriever' "
+                "AND sparse_retriever_id IS NOT NULL "
+                "AND dense_retriever_id IS NULL) OR "
+                "(class_name = 'DenseRetriever' "
+                "AND dense_retriever_id IS NOT NULL "
+                "AND sparse_retriever_id IS NULL)"
+            )
+        )

DashAI/alembic/versions/b2c3d4e5f6a7_add_uniqueness_constraints_rag.py

@@ -0,0 +1,97 @@
+"""add uniqueness constraints to rag tables and fix storage tracing
+
+Revision ID: b2c3d4e5f6a7
+Revises: a1b2c3d4e5f6
+Create Date: 2026-05-25 12:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = 'b2c3d4e5f6a7'
+down_revision: Union[str, None] = 'a1b2c3d4e5f6'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ── Chunk: replace useless (id, document_id) with (document_id, chunk_index, chunking_model_id) ──
+    with op.batch_alter_table('chunk', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_chunk_document', type_='unique')
+        batch_op.create_unique_constraint(
+            'uix_document_chunk_index_chunking',
+            ['document_id', 'chunk_index', 'chunking_model_id'],
+        )
+
+    # ── RAGPrompt ──
+    with op.batch_alter_table('rag_prompt', schema=None) as batch_op:
+        batch_op.create_unique_constraint(
+            'uix_rag_prompt_class_params',
+            ['class_name', 'parameters'],
+        )
+
+    # ── RAGGenerationModel ──
+    with op.batch_alter_table('rag_generation_model', schema=None) as batch_op:
+        batch_op.create_unique_constraint(
+            'uix_rag_gen_model_class_params',
+            ['class_name', 'parameters'],
+        )
+
+    # ── RAGChunkingModel ──
+    with op.batch_alter_table('rag_chunking_model', schema=None) as batch_op:
+        batch_op.create_unique_constraint(
+            'uix_rag_chunking_model_class_params',
+            ['class_name', 'parameters'],
+        )
+
+    # ── RAGEmbeddingModel ──
+    with op.batch_alter_table('rag_embedding_model', schema=None) as batch_op:
+        batch_op.create_unique_constraint(
+            'uix_rag_embedding_model_class_params',
+            ['class_name', 'parameters'],
+        )
+
+    # ── RAGSparseRetriever ──
+    with op.batch_alter_table('rag_sparse_retriever', schema=None) as batch_op:
+        batch_op.create_unique_constraint(
+            'uix_rag_sparse_retriever',
+            ['class_name', 'parameters', 'documents_ids', 'chunking_model_id'],
+        )
+
+    # ── RAGDenseRetriever ──
+    with op.batch_alter_table('rag_dense_retriever', schema=None) as batch_op:
+        batch_op.create_unique_constraint(
+            'uix_rag_dense_retriever',
+            ['class_name', 'parameters', 'document_ids',
+             'chunking_model_id', 'embedding_model_id'],
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table('rag_dense_retriever', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_rag_dense_retriever', type_='unique')
+
+    with op.batch_alter_table('rag_sparse_retriever', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_rag_sparse_retriever', type_='unique')
+
+    with op.batch_alter_table('rag_embedding_model', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_rag_embedding_model_class_params', type_='unique')
+
+    with op.batch_alter_table('rag_chunking_model', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_rag_chunking_model_class_params', type_='unique')
+
+    with op.batch_alter_table('rag_generation_model', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_rag_gen_model_class_params', type_='unique')
+
+    with op.batch_alter_table('rag_prompt', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_rag_prompt_class_params', type_='unique')
+
+    with op.batch_alter_table('chunk', schema=None) as batch_op:
+        batch_op.drop_constraint('uix_document_chunk_index_chunking', type_='unique')
+        batch_op.create_unique_constraint(
+            'uix_chunk_document',
+            ['id', 'document_id'],
+        )