Refactor Docs, Clean Codebase, and Improve Config Consistency

auto-analyst-backend/docs/README.md

@@ -17,7 +17,7 @@ This directory contains comprehensive documentation for the Auto-Analyst backend
 ### **🌐 API** (`/api/`)
 - **[API Endpoints Overview](./api/endpoints.md)** - Main API reference hub
 - **[Route Documentation](./api/routes/)** - Detailed endpoint documentation:
-  - **[Core Routes](./api/routes/core.md)** - File uploads, sessions, authentication
+  - **[Core Routes](./api/routes/session.md)** - File uploads, sessions, authentication
   - **[Chat Routes](./api/routes/chats.md)** - Chat and messaging endpoints
   - **[Code Routes](./api/routes/code.md)** - Code execution and processing
   - **[Analytics Routes](./api/routes/analytics.md)** - Usage analytics and monitoring
@@ -105,9 +105,9 @@ FRONTEND_URL=http://localhost:3000/
 ```bash
 # Initialize database and default agents
 python -c "
-from src.db.init_db import init_database
+from src.db.init_db import init_db
 from src.db.init_default_agents import initialize_default_agents
-init_database()
+init_db()
 initialize_default_agents()
 print('✅ Database and agents initialized successfully')
 "
@@ -117,7 +117,7 @@ print('✅ Database and agents initialized successfully')
 
 ```bash
 # Start the FastAPI server
-python app.py
+python -m app
 
 # Or with uvicorn for more control
 uvicorn app:app --reload --host 0.0.0.0 --port 8000
@@ -127,7 +127,6 @@ uvicorn app:app --reload --host 0.0.0.0 --port 8000
 
 - **API Documentation**: `http://localhost:8000/docs`
 - **Health Check**: `http://localhost:8000/health`
-- **Interactive API**: `http://localhost:8000/redoc`
 
 ## 🔧 Development Workflow
 

auto-analyst-backend/docs/api/endpoints.md

@@ -14,7 +14,7 @@ For more specific details regarding the various functionalities offered by the A
 
 ## 🛠️ API Reference
 
-- **[Core Endpoints](./routes/core.md)**: Review the core endpoints that handle fundamental operations within the application, including data uploads, AI analysis, model settings, and session management.
+- **[Core Endpoints](./routes/session.md)**: Review the core endpoints that handle fundamental operations within the application, including data uploads, AI analysis, model settings, and session management.
 - **[Analytics Endpoints](./routes/analytics.md)**: Explore the endpoints dedicated to analytics, providing insights into usage statistics, performance metrics, cost analysis, and real-time monitoring.
 - **[Chat Endpoints](./routes/chats.md)**: Discover the endpoints that manage chat interactions, enabling users to create, retrieve, and manage chat sessions effectively.
 - **[Code Endpoints](./routes/code.md)**: Learn about the endpoints for code execution, editing, fixing, and cleaning operations with advanced AI assistance.

auto-analyst-backend/docs/api/routes/templates.md

@@ -54,7 +54,7 @@ The system includes four core default agents that are **enabled by default** for
     "icon_url": "/icons/templates/preprocessing_agent.svg",
     "is_premium_only": false,
     "is_active": true,
-    "usage_count": 1234,
+    "usage_count": 12,
     "created_at": "2023-05-01T12:00:00Z",
     "updated_at": "2023-05-01T12:00:00Z"
   }

auto-analyst-backend/docs/architecture/architecture.md

@@ -141,14 +141,16 @@ Users (1) ──────── (Many) Chats
 
 | Module | Purpose | Key Endpoints |
 |--------|---------|---------------|
-| `core_routes.py` | Core functionality | `/upload_excel`, `/session_info`, `/health` |
+| `session_routes.py` | Core functionality | `/upload_excel`, `/session_info` |
 | `chat_routes.py` | Chat management | `/chats`, `/messages`, `/delete_chat` |
 | `code_routes.py` | Code operations | `/execute_code`, `/get_latest_code` |
 | `templates_routes.py` | Agent templates | `/templates`, `/user/{id}/enabled` |
 | `deep_analysis_routes.py` | Deep analysis | `/reports`, `/download_from_db` |
 | `analytics_routes.py` | System analytics | `/usage`, `/feedback`, `/costs` |
 | `feedback_routes.py` | User feedback | `/feedback`, `/message/{id}/feedback` |
 
+NOTE: Make sure to add a router prefix when calling these endpoints, such as to get dashboard, you'll use `http://localhost:8000/templates/dashboard`
+
 ### 5. Business Logic Layer (`src/managers/`)
 
 **Service Layer for Complex Operations**
@@ -239,11 +241,6 @@ Capability Mapping → Execution Routing → Usage Tracking
 - Event-driven model updates
 - Real-time progress notifications
 
-### 5. **Factory Pattern**
-- Agent creation based on template configurations
-- Session factory for database connections
-- Dynamic model instantiation
-
 ## 🔧 Configuration Management
 
 ### Environment Configuration
@@ -399,7 +396,7 @@ FROM python:3.11-slim as base
 
 ### Third-Party Integration
 
-1. **Python Data Science Stack**:
+1. **Python Data Science Stack (For Agentic Use only)**:
    - Pandas for data manipulation
    - NumPy for numerical computing
    - Scikit-learn for machine learning

auto-analyst-backend/docs/development/development_workflow.md

@@ -28,7 +28,7 @@ src/
 │   ├── ai_manager.py      # AI model management
 │   └── session_manager.py # Session lifecycle
 ├── routes/          # FastAPI route handlers
-│   ├── core_routes.py     # Core functionality
+│   ├── session_routes.py     # Core functionality
 │   ├── chat_routes.py     # Chat endpoints
 │   └── [feature]_routes.py # Feature-specific routes
 ├── utils/           # Shared utilities
@@ -89,7 +89,7 @@ class new_analysis_agent(dspy.Signature):
   "template_name": "new_analysis_agent",
   "description": "Performs specialized analysis on datasets",
   "variant_type": "both",  # individual, planner, or both
-  "is_premium": false,
+  "is_premium": false, # Will be active by default
   "usage_count": 0,
   "icon_url": "analysis.svg"
 }
@@ -390,89 +390,6 @@ async def async_database_operation(session: Session) -> Any:
         session.close()
 ```
 
-## 🧪 Testing Patterns
-
-### 1. **Unit Testing Structure**
-
-```python
-# tests/test_feature_manager.py
-import pytest
-from unittest.mock import Mock, patch
-from src.managers.feature_manager import FeatureManager
-from src.db.schemas.models import FeatureModel
-
-class TestFeatureManager:
-    """Test cases for FeatureManager class."""
-
-    @pytest.fixture
-    def mock_session(self):
-        """Create a mock database session."""
-        return Mock()
-
-    @pytest.fixture
-    def feature_manager(self, mock_session):
-        """Create FeatureManager instance with mock session."""
-        return FeatureManager(mock_session)
-
-    async def test_create_feature_success(self, feature_manager, mock_session):
-        """Test successful feature creation."""
-        # Arrange
-        mock_session.query.return_value.filter_by.return_value.first.return_value = None
-
-        # Act
-        result = await feature_manager.create_feature("test_feature", "Test description")
-
-        # Assert
-        assert isinstance(result, FeatureModel)
-        mock_session.add.assert_called_once()
-        mock_session.commit.assert_called_once()
-
-    async def test_create_feature_duplicate_name(self, feature_manager, mock_session):
-        """Test feature creation with duplicate name."""
-        # Arrange
-        existing_feature = FeatureModel(name="test_feature")
-        mock_session.query.return_value.filter_by.return_value.first.return_value = existing_feature
-
-        # Act & Assert
-        with pytest.raises(ValueError, match="already exists"):
-            await feature_manager.create_feature("test_feature")
-```
-
-### 2. **Integration Testing**
-
-```python
-# tests/integration/test_api_endpoints.py
-import pytest
-from fastapi.testclient import TestClient
-from src.app import app
-from src.db.init_db import session_factory
-
-client = TestClient(app)
-
-class TestFeatureAPI:
-    """Integration tests for feature API endpoints."""
-
-    def test_create_feature_endpoint(self):
-        """Test feature creation through API."""
-        response = client.post(
-            "/feature/",
-            json={"name": "test_feature", "description": "Test description"}
-        )
-
-        assert response.status_code == 200
-        data = response.json()
-        assert data["name"] == "test_feature"
-        assert "id" in data
-
-    def test_get_features_endpoint(self):
-        """Test feature retrieval through API."""
-        response = client.get("/feature/")
-
-        assert response.status_code == 200
-        data = response.json()
-        assert isinstance(data, list)
-```
-
 ## 🔧 Development Workflow
 
 ### 1. **Feature Development Process**
@@ -501,37 +418,6 @@ class TestFeatureAPI:
    alembic upgrade head
    ```
 
-4. **Testing**:
-   ```bash
-   # Run unit tests
-   pytest tests/unit/
-
-   # Run integration tests
-   pytest tests/integration/
-
-   # Test API endpoints
-   curl -X POST "http://localhost:8000/feature/test"
-   ```
-
-5. **Documentation**:
-   ```bash
-   # Update API documentation
-   # Add to troubleshooting guide if needed
-   # Update getting started guide
-   ```
-
-### 2. **Code Review Checklist**
-
-- [ ] **Type Hints**: All functions have proper type annotations
-- [ ] **Documentation**: Docstrings for all public functions and classes
-- [ ] **Error Handling**: Comprehensive exception handling with logging
-- [ ] **Testing**: Unit tests cover all new functionality
-- [ ] **Database**: Proper session management and transaction handling
-- [ ] **Async**: Appropriate use of async/await patterns
-- [ ] **Logging**: Meaningful log messages at appropriate levels
-- [ ] **Security**: Input validation and authorization checks
-- [ ] **Performance**: Efficient database queries and memory usage
-
 ### 3. **Release Process**
 
 1. **Pre-release Testing**:
@@ -561,8 +447,6 @@ class TestFeatureAPI:
    # Test container build
    docker build -t auto-analyst-backend .
 
-   # Verify environment variables
-   python scripts/verify_env.py
    ```
 
 ## 📊 Performance Considerations
@@ -589,28 +473,8 @@ def get_paginated_results(session, model, page=1, per_page=20):
     return session.query(model).offset(offset).limit(per_page).all()
 ```
 
-### 2. **Memory Management**
-
-```python
-# Efficient data processing
-def process_large_dataset(file_path: str):
-    # Bad: Load entire file into memory
-    # df = pd.read_csv(file_path)
-
-    # Good: Process in chunks
-    chunk_size = 1000
-    for chunk in pd.read_csv(file_path, chunksize=chunk_size):
-        process_chunk(chunk)
-        # Memory is freed after each chunk
-
-# Clean up resources
-def cleanup_session_data(session_id: str):
-    if session_id in global_session_data:
-        del global_session_data[session_id]
-    gc.collect()  # Force garbage collection
-```
 
-### 3. **Async Optimization**
+### 2. **Async Optimization**
 
 ```python
 # Use connection pooling

auto-analyst-backend/docs/getting_started.md

@@ -69,9 +69,9 @@ ADMIN_API_KEY=your_admin_key_here
 ```bash
 # Initialize database and default agents
 python -c "
-from src.db.init_db import init_database
+from src.db.init_db import init_db
 from src.db.init_default_agents import initialize_default_agents
-init_database()
+init_db()
 initialize_default_agents()
 print('✅ Database initialized successfully')
 "
@@ -103,7 +103,7 @@ Visit: `http://localhost:8000/docs` for interactive API documentation
 
 ### Route Files (API Endpoints)
 
-- **`src/routes/core_routes.py`** - File uploads, sessions, authentication
+- **`src/routes/session_routes.py`** - File uploads, sessions, authentication
 - **`src/routes/chat_routes.py`** - Chat and messaging
 - **`src/routes/code_routes.py`** - Code execution and processing
 - **`src/routes/templates_routes.py`** - Agent template management

auto-analyst-backend/docs/troubleshooting/troubleshooting.md

@@ -13,9 +13,9 @@
 1. **Initialize Database**:
    ```bash
    python -c "
-   from src.db.init_db import init_database
+   from src.db.init_db import init_db
    from src.db.init_default_agents import initialize_default_agents
-   init_database()
+   init_db()
    initialize_default_agents()
    print('✅ Database initialized')
    "
@@ -305,7 +305,7 @@
    ```bash
    # For SQLite (development)
    rm auto_analyst.db
-   python -c "from src.db.init_db import init_database; init_database()"
+   python -c "from src.db.init_db import init_db; init_db()"
    ```
 
 #### Problem: Constraint Violations

auto-analyst-backend/src/managers/ai_manager.py

@@ -1,10 +1,8 @@
 import logging
 from typing import Optional, Dict, Any
-import time
 from src.db.schemas.models import ModelUsage
 from src.db.init_db import session_factory
 from datetime import datetime, UTC
-import tiktoken
 from src.routes.analytics_routes import handle_new_model_usage
 import asyncio
 

auto-analyst-backend/src/managers/chat_manager.py

@@ -1,14 +1,10 @@
-from sqlalchemy import create_engine, desc, func, exists
+from sqlalchemy import create_engine, func, exists
 from sqlalchemy.orm import sessionmaker, scoped_session
 from sqlalchemy.exc import SQLAlchemyError
 from src.db.schemas.models import Base, User, Chat, Message, ModelUsage, MessageFeedback
 import logging
-import requests
-import json
-from typing import List, Dict, Optional, Tuple, Any
+from typing import List, Dict, Optional, Any
 from datetime import datetime, UTC
-import time
-import tiktoken
 from src.utils.logger import Logger
 import re
 

auto-analyst-backend/src/managers/session_manager.py

@@ -4,12 +4,12 @@
 import uuid
 import logging
 import pandas as pd
-from typing import Dict, Any, List, Optional
+from typing import Dict, Any, List
 
 from llama_index.core import Document, VectorStoreIndex
 from src.utils.logger import Logger
-from src.managers.user_manager import create_user, get_current_user, get_user_by_email
-from src.agents.agents import auto_analyst, auto_analyst_ind
+from src.managers.user_manager import get_current_user
+from src.agents.agents import auto_analyst
 from src.agents.retrievers.retrievers import make_data
 from src.managers.chat_manager import ChatManager
 from dotenv import load_dotenv

auto-analyst-backend/src/routes/templates_routes.py

@@ -736,7 +736,7 @@ async def get_templates_by_category(category: str):
                 icon_url=template.icon_url,
                 is_premium_only=template.is_premium_only,
                 is_active=template.is_active,
-                usage_count=global_usage.get(template.template_id, 0),  # Global usage count
+                usage_count=global_usage.get(template.template_id, 0),  # Global usage count (shows how many times this template has been used overall by users)
                 created_at=template.created_at,
                 updated_at=template.updated_at
             ) for template in templates]