diff --git a/auto-analyst-backend/docs/README.md b/auto-analyst-backend/docs/README.md index f50cf5b6..03f5483f 100644 --- a/auto-analyst-backend/docs/README.md +++ b/auto-analyst-backend/docs/README.md @@ -17,7 +17,7 @@ This directory contains comprehensive documentation for the Auto-Analyst backend ### **๐ŸŒ API** (`/api/`) - **[API Endpoints Overview](./api/endpoints.md)** - Main API reference hub - **[Route Documentation](./api/routes/)** - Detailed endpoint documentation: - - **[Core Routes](./api/routes/core.md)** - File uploads, sessions, authentication + - **[Core Routes](./api/routes/session.md)** - File uploads, sessions, authentication - **[Chat Routes](./api/routes/chats.md)** - Chat and messaging endpoints - **[Code Routes](./api/routes/code.md)** - Code execution and processing - **[Analytics Routes](./api/routes/analytics.md)** - Usage analytics and monitoring @@ -105,9 +105,9 @@ FRONTEND_URL=http://localhost:3000/ ```bash # Initialize database and default agents python -c " -from src.db.init_db import init_database +from src.db.init_db import init_db from src.db.init_default_agents import initialize_default_agents -init_database() +init_db() initialize_default_agents() print('โœ… Database and agents initialized successfully') " @@ -117,7 +117,7 @@ print('โœ… Database and agents initialized successfully') ```bash # Start the FastAPI server -python app.py +python -m app # Or with uvicorn for more control uvicorn app:app --reload --host 0.0.0.0 --port 8000 @@ -127,7 +127,6 @@ uvicorn app:app --reload --host 0.0.0.0 --port 8000 - **API Documentation**: `http://localhost:8000/docs` - **Health Check**: `http://localhost:8000/health` -- **Interactive API**: `http://localhost:8000/redoc` ## ๐Ÿ”ง Development Workflow diff --git a/auto-analyst-backend/docs/api/endpoints.md b/auto-analyst-backend/docs/api/endpoints.md index 3838508a..1738d933 100644 --- a/auto-analyst-backend/docs/api/endpoints.md +++ b/auto-analyst-backend/docs/api/endpoints.md @@ -14,7 +14,7 @@ For more specific details regarding the various functionalities offered by the A ## ๐Ÿ› ๏ธ API Reference -- **[Core Endpoints](./routes/core.md)**: Review the core endpoints that handle fundamental operations within the application, including data uploads, AI analysis, model settings, and session management. +- **[Core Endpoints](./routes/session.md)**: Review the core endpoints that handle fundamental operations within the application, including data uploads, AI analysis, model settings, and session management. - **[Analytics Endpoints](./routes/analytics.md)**: Explore the endpoints dedicated to analytics, providing insights into usage statistics, performance metrics, cost analysis, and real-time monitoring. - **[Chat Endpoints](./routes/chats.md)**: Discover the endpoints that manage chat interactions, enabling users to create, retrieve, and manage chat sessions effectively. - **[Code Endpoints](./routes/code.md)**: Learn about the endpoints for code execution, editing, fixing, and cleaning operations with advanced AI assistance. diff --git a/auto-analyst-backend/docs/api/routes/core.md b/auto-analyst-backend/docs/api/routes/session.md similarity index 100% rename from auto-analyst-backend/docs/api/routes/core.md rename to auto-analyst-backend/docs/api/routes/session.md diff --git a/auto-analyst-backend/docs/api/routes/templates.md b/auto-analyst-backend/docs/api/routes/templates.md index e63febb4..29803b79 100644 --- a/auto-analyst-backend/docs/api/routes/templates.md +++ b/auto-analyst-backend/docs/api/routes/templates.md @@ -54,7 +54,7 @@ The system includes four core default agents that are **enabled by default** for "icon_url": "/icons/templates/preprocessing_agent.svg", "is_premium_only": false, "is_active": true, - "usage_count": 1234, + "usage_count": 12, "created_at": "2023-05-01T12:00:00Z", "updated_at": "2023-05-01T12:00:00Z" } diff --git a/auto-analyst-backend/docs/architecture/architecture.md b/auto-analyst-backend/docs/architecture/architecture.md index 6bce7d14..01d6dcb4 100644 --- a/auto-analyst-backend/docs/architecture/architecture.md +++ b/auto-analyst-backend/docs/architecture/architecture.md @@ -141,7 +141,7 @@ Users (1) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ (Many) Chats | Module | Purpose | Key Endpoints | |--------|---------|---------------| -| `core_routes.py` | Core functionality | `/upload_excel`, `/session_info`, `/health` | +| `session_routes.py` | Core functionality | `/upload_excel`, `/session_info` | | `chat_routes.py` | Chat management | `/chats`, `/messages`, `/delete_chat` | | `code_routes.py` | Code operations | `/execute_code`, `/get_latest_code` | | `templates_routes.py` | Agent templates | `/templates`, `/user/{id}/enabled` | @@ -149,6 +149,8 @@ Users (1) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ (Many) Chats | `analytics_routes.py` | System analytics | `/usage`, `/feedback`, `/costs` | | `feedback_routes.py` | User feedback | `/feedback`, `/message/{id}/feedback` | +NOTE: Make sure to add a router prefix when calling these endpoints, such as to get dashboard, you'll use `http://localhost:8000/templates/dashboard` + ### 5. Business Logic Layer (`src/managers/`) **Service Layer for Complex Operations** @@ -239,11 +241,6 @@ Capability Mapping โ†’ Execution Routing โ†’ Usage Tracking - Event-driven model updates - Real-time progress notifications -### 5. **Factory Pattern** -- Agent creation based on template configurations -- Session factory for database connections -- Dynamic model instantiation - ## ๐Ÿ”ง Configuration Management ### Environment Configuration @@ -399,7 +396,7 @@ FROM python:3.11-slim as base ### Third-Party Integration -1. **Python Data Science Stack**: +1. **Python Data Science Stack (For Agentic Use only)**: - Pandas for data manipulation - NumPy for numerical computing - Scikit-learn for machine learning diff --git a/auto-analyst-backend/docs/development/development_workflow.md b/auto-analyst-backend/docs/development/development_workflow.md index 10736f40..6b3852de 100644 --- a/auto-analyst-backend/docs/development/development_workflow.md +++ b/auto-analyst-backend/docs/development/development_workflow.md @@ -28,7 +28,7 @@ src/ โ”‚ โ”œโ”€โ”€ ai_manager.py # AI model management โ”‚ โ””โ”€โ”€ session_manager.py # Session lifecycle โ”œโ”€โ”€ routes/ # FastAPI route handlers -โ”‚ โ”œโ”€โ”€ core_routes.py # Core functionality +โ”‚ โ”œโ”€โ”€ session_routes.py # Core functionality โ”‚ โ”œโ”€โ”€ chat_routes.py # Chat endpoints โ”‚ โ””โ”€โ”€ [feature]_routes.py # Feature-specific routes โ”œโ”€โ”€ utils/ # Shared utilities @@ -89,7 +89,7 @@ class new_analysis_agent(dspy.Signature): "template_name": "new_analysis_agent", "description": "Performs specialized analysis on datasets", "variant_type": "both", # individual, planner, or both - "is_premium": false, + "is_premium": false, # Will be active by default "usage_count": 0, "icon_url": "analysis.svg" } @@ -390,89 +390,6 @@ async def async_database_operation(session: Session) -> Any: session.close() ``` -## ๐Ÿงช Testing Patterns - -### 1. **Unit Testing Structure** - -```python -# tests/test_feature_manager.py -import pytest -from unittest.mock import Mock, patch -from src.managers.feature_manager import FeatureManager -from src.db.schemas.models import FeatureModel - -class TestFeatureManager: - """Test cases for FeatureManager class.""" - - @pytest.fixture - def mock_session(self): - """Create a mock database session.""" - return Mock() - - @pytest.fixture - def feature_manager(self, mock_session): - """Create FeatureManager instance with mock session.""" - return FeatureManager(mock_session) - - async def test_create_feature_success(self, feature_manager, mock_session): - """Test successful feature creation.""" - # Arrange - mock_session.query.return_value.filter_by.return_value.first.return_value = None - - # Act - result = await feature_manager.create_feature("test_feature", "Test description") - - # Assert - assert isinstance(result, FeatureModel) - mock_session.add.assert_called_once() - mock_session.commit.assert_called_once() - - async def test_create_feature_duplicate_name(self, feature_manager, mock_session): - """Test feature creation with duplicate name.""" - # Arrange - existing_feature = FeatureModel(name="test_feature") - mock_session.query.return_value.filter_by.return_value.first.return_value = existing_feature - - # Act & Assert - with pytest.raises(ValueError, match="already exists"): - await feature_manager.create_feature("test_feature") -``` - -### 2. **Integration Testing** - -```python -# tests/integration/test_api_endpoints.py -import pytest -from fastapi.testclient import TestClient -from src.app import app -from src.db.init_db import session_factory - -client = TestClient(app) - -class TestFeatureAPI: - """Integration tests for feature API endpoints.""" - - def test_create_feature_endpoint(self): - """Test feature creation through API.""" - response = client.post( - "/feature/", - json={"name": "test_feature", "description": "Test description"} - ) - - assert response.status_code == 200 - data = response.json() - assert data["name"] == "test_feature" - assert "id" in data - - def test_get_features_endpoint(self): - """Test feature retrieval through API.""" - response = client.get("/feature/") - - assert response.status_code == 200 - data = response.json() - assert isinstance(data, list) -``` - ## ๐Ÿ”ง Development Workflow ### 1. **Feature Development Process** @@ -501,37 +418,6 @@ class TestFeatureAPI: alembic upgrade head ``` -4. **Testing**: - ```bash - # Run unit tests - pytest tests/unit/ - - # Run integration tests - pytest tests/integration/ - - # Test API endpoints - curl -X POST "http://localhost:8000/feature/test" - ``` - -5. **Documentation**: - ```bash - # Update API documentation - # Add to troubleshooting guide if needed - # Update getting started guide - ``` - -### 2. **Code Review Checklist** - -- [ ] **Type Hints**: All functions have proper type annotations -- [ ] **Documentation**: Docstrings for all public functions and classes -- [ ] **Error Handling**: Comprehensive exception handling with logging -- [ ] **Testing**: Unit tests cover all new functionality -- [ ] **Database**: Proper session management and transaction handling -- [ ] **Async**: Appropriate use of async/await patterns -- [ ] **Logging**: Meaningful log messages at appropriate levels -- [ ] **Security**: Input validation and authorization checks -- [ ] **Performance**: Efficient database queries and memory usage - ### 3. **Release Process** 1. **Pre-release Testing**: @@ -561,8 +447,6 @@ class TestFeatureAPI: # Test container build docker build -t auto-analyst-backend . - # Verify environment variables - python scripts/verify_env.py ``` ## ๐Ÿ“Š Performance Considerations @@ -589,28 +473,8 @@ def get_paginated_results(session, model, page=1, per_page=20): return session.query(model).offset(offset).limit(per_page).all() ``` -### 2. **Memory Management** - -```python -# Efficient data processing -def process_large_dataset(file_path: str): - # Bad: Load entire file into memory - # df = pd.read_csv(file_path) - - # Good: Process in chunks - chunk_size = 1000 - for chunk in pd.read_csv(file_path, chunksize=chunk_size): - process_chunk(chunk) - # Memory is freed after each chunk - -# Clean up resources -def cleanup_session_data(session_id: str): - if session_id in global_session_data: - del global_session_data[session_id] - gc.collect() # Force garbage collection -``` -### 3. **Async Optimization** +### 2. **Async Optimization** ```python # Use connection pooling diff --git a/auto-analyst-backend/docs/getting_started.md b/auto-analyst-backend/docs/getting_started.md index 86894fbe..69eaff77 100644 --- a/auto-analyst-backend/docs/getting_started.md +++ b/auto-analyst-backend/docs/getting_started.md @@ -69,9 +69,9 @@ ADMIN_API_KEY=your_admin_key_here ```bash # Initialize database and default agents python -c " -from src.db.init_db import init_database +from src.db.init_db import init_db from src.db.init_default_agents import initialize_default_agents -init_database() +init_db() initialize_default_agents() print('โœ… Database initialized successfully') " @@ -103,7 +103,7 @@ Visit: `http://localhost:8000/docs` for interactive API documentation ### Route Files (API Endpoints) -- **`src/routes/core_routes.py`** - File uploads, sessions, authentication +- **`src/routes/session_routes.py`** - File uploads, sessions, authentication - **`src/routes/chat_routes.py`** - Chat and messaging - **`src/routes/code_routes.py`** - Code execution and processing - **`src/routes/templates_routes.py`** - Agent template management diff --git a/auto-analyst-backend/docs/troubleshooting/troubleshooting.md b/auto-analyst-backend/docs/troubleshooting/troubleshooting.md index 4152e3a1..5c111517 100644 --- a/auto-analyst-backend/docs/troubleshooting/troubleshooting.md +++ b/auto-analyst-backend/docs/troubleshooting/troubleshooting.md @@ -13,9 +13,9 @@ 1. **Initialize Database**: ```bash python -c " - from src.db.init_db import init_database + from src.db.init_db import init_db from src.db.init_default_agents import initialize_default_agents - init_database() + init_db() initialize_default_agents() print('โœ… Database initialized') " @@ -305,7 +305,7 @@ ```bash # For SQLite (development) rm auto_analyst.db - python -c "from src.db.init_db import init_database; init_database()" + python -c "from src.db.init_db import init_db; init_db()" ``` #### Problem: Constraint Violations diff --git a/auto-analyst-backend/src/managers/ai_manager.py b/auto-analyst-backend/src/managers/ai_manager.py index c59d1c61..6446ba29 100644 --- a/auto-analyst-backend/src/managers/ai_manager.py +++ b/auto-analyst-backend/src/managers/ai_manager.py @@ -1,10 +1,8 @@ import logging from typing import Optional, Dict, Any -import time from src.db.schemas.models import ModelUsage from src.db.init_db import session_factory from datetime import datetime, UTC -import tiktoken from src.routes.analytics_routes import handle_new_model_usage import asyncio diff --git a/auto-analyst-backend/src/managers/chat_manager.py b/auto-analyst-backend/src/managers/chat_manager.py index 8c304c23..4b602a3a 100644 --- a/auto-analyst-backend/src/managers/chat_manager.py +++ b/auto-analyst-backend/src/managers/chat_manager.py @@ -1,14 +1,10 @@ -from sqlalchemy import create_engine, desc, func, exists +from sqlalchemy import create_engine, func, exists from sqlalchemy.orm import sessionmaker, scoped_session from sqlalchemy.exc import SQLAlchemyError from src.db.schemas.models import Base, User, Chat, Message, ModelUsage, MessageFeedback import logging -import requests -import json -from typing import List, Dict, Optional, Tuple, Any +from typing import List, Dict, Optional, Any from datetime import datetime, UTC -import time -import tiktoken from src.utils.logger import Logger import re diff --git a/auto-analyst-backend/src/managers/session_manager.py b/auto-analyst-backend/src/managers/session_manager.py index 9ccbd292..37122ebc 100644 --- a/auto-analyst-backend/src/managers/session_manager.py +++ b/auto-analyst-backend/src/managers/session_manager.py @@ -4,12 +4,12 @@ import uuid import logging import pandas as pd -from typing import Dict, Any, List, Optional +from typing import Dict, Any, List from llama_index.core import Document, VectorStoreIndex from src.utils.logger import Logger -from src.managers.user_manager import create_user, get_current_user, get_user_by_email -from src.agents.agents import auto_analyst, auto_analyst_ind +from src.managers.user_manager import get_current_user +from src.agents.agents import auto_analyst from src.agents.retrievers.retrievers import make_data from src.managers.chat_manager import ChatManager from dotenv import load_dotenv diff --git a/auto-analyst-backend/src/routes/templates_routes.py b/auto-analyst-backend/src/routes/templates_routes.py index e27c3ae2..0d69f7cd 100644 --- a/auto-analyst-backend/src/routes/templates_routes.py +++ b/auto-analyst-backend/src/routes/templates_routes.py @@ -736,7 +736,7 @@ async def get_templates_by_category(category: str): icon_url=template.icon_url, is_premium_only=template.is_premium_only, is_active=template.is_active, - usage_count=global_usage.get(template.template_id, 0), # Global usage count + usage_count=global_usage.get(template.template_id, 0), # Global usage count (shows how many times this template has been used overall by users) created_at=template.created_at, updated_at=template.updated_at ) for template in templates] diff --git a/auto-analyst-frontend/app/api/webhooks/route.ts b/auto-analyst-frontend/app/api/webhooks/route.ts index c1ab6a88..0360f306 100644 --- a/auto-analyst-frontend/app/api/webhooks/route.ts +++ b/auto-analyst-frontend/app/api/webhooks/route.ts @@ -4,6 +4,7 @@ import { Readable } from 'stream' import redis, { creditUtils, KEYS, profileUtils } from '@/lib/redis' import { sendSubscriptionConfirmation, sendPaymentConfirmationEmail } from '@/lib/email' import logger from '@/lib/utils/logger' +import { headers } from 'next/headers' import { CreditConfig } from '@/lib/credits-config' // Use the correct App Router configuration instead of the default body parser @@ -18,122 +19,6 @@ const stripe = process.env.STRIPE_SECRET_KEY const webhookSecret = process.env.STRIPE_WEBHOOK_SECRET || '' -// Helper function to read the raw request body as text -async function getRawBody(readable: Readable): Promise { - const chunks: Buffer[] = [] - for await (const chunk of readable) { - chunks.push(typeof chunk === 'string' ? Buffer.from(chunk) : chunk) - } - return Buffer.concat(chunks) -} - -// Helper function to update a user's subscription information -async function updateUserSubscription(userId: string, session: Stripe.Checkout.Session) { - try { - // Retrieve the complete line items to get product details - const lineItems = await stripe!.checkout.sessions.listLineItems(session.id) - if (!lineItems.data.length) return false - - // Get the price ID from the line item - const priceId = lineItems.data[0].price?.id - if (!priceId) return false - - // Retrieve price to get recurring interval and product ID - const price = await stripe!.prices.retrieve(priceId) - - // Retrieve product details - const product = await stripe!.products.retrieve(price.product as string) - - // Extract subscription details - const planName = product.name - let interval = 'month' - if (planName.toLowerCase().includes('yearly') || price?.recurring?.interval === 'year') { - interval = 'year' - } else if (planName.toLowerCase().includes('daily') || price?.recurring?.interval === 'day') { - interval = 'day' - } - const amount = price.unit_amount! / 100 // Convert from cents to dollars - - // Calculate next renewal date - const now = new Date() - let renewalDate = new Date() - if (interval === 'month') { - renewalDate.setMonth(now.getMonth() + 1) - } else if (interval === 'year') { - renewalDate.setFullYear(now.getFullYear() + 1) - } else if (interval === 'day') { - renewalDate.setDate(now.getDate() + 1) - } - - // Determine credit amounts based on plan using centralized config - const planCredits = CreditConfig.getCreditsForPlan(planName) - const creditAmount = planCredits.total - const planType = planCredits.type - - // Update subscription data - await redis.hset(KEYS.USER_SUBSCRIPTION(userId), { - plan: planName, - planType, - status: 'active', - amount: amount.toString(), - interval, - purchaseDate: now.toISOString(), - renewalDate: renewalDate.toISOString().split('T')[0], - lastUpdated: now.toISOString(), - stripeCustomerId: session.customer || '', - stripeSubscriptionId: session.subscription || '' - }) - - // Update credit information - let resetDate = creditUtils.getNextMonthFirstDay(); - - // For daily plans, set reset date to tomorrow - if (interval === 'day') { - const tomorrow = new Date(); - tomorrow.setDate(tomorrow.getDate() + 1); - resetDate = tomorrow.toISOString().split('T')[0]; - } - - await redis.hset(KEYS.USER_CREDITS(userId), { - total: creditAmount.toString(), - used: '0', - resetDate: resetDate, - lastUpdate: now.toISOString() - }) - - // Get user email from session or lookup in Redis - let userEmail = session.customer_email || '' - if (!userEmail && userId) { - // Try to fetch email from Redis user profile - const userProfile = await profileUtils.getUserProfile(userId) - if (userProfile && userProfile.email) { - userEmail = userProfile.email as string - } - } - - // Send confirmation email if we have the user's email - if (userEmail) { - await sendSubscriptionConfirmation( - userEmail, - planName, - planType, - amount, - interval, - renewalDate.toISOString().split('T')[0], - creditAmount, - resetDate - ) - } else { - logger.log(`No email found for user ${userId}, cannot send confirmation email`) - } - - return true - } catch (error) { - console.error('Error updating user subscription:', error) - return false - } -} - export async function POST(request: NextRequest) { try { // Check if Stripe is initialized @@ -148,31 +33,24 @@ export async function POST(request: NextRequest) { return NextResponse.json({ error: 'Webhook secret not configured' }, { status: 500 }) } - const signature = request.headers.get('stripe-signature') + const signature = headers().get('Stripe-Signature') as string if (!signature) { console.error('โŒ No Stripe signature found in request headers') return NextResponse.json({ error: 'No Stripe signature found' }, { status: 400 }) } - // Get the raw request body - let rawBody: Buffer - try { - rawBody = await getRawBody(request.body as unknown as Readable) - } catch (bodyError) { - console.error('โŒ Failed to read webhook body:', bodyError) - return NextResponse.json({ error: 'Failed to read request body' }, { status: 400 }) - } + const body = await request.text() // Verify the webhook signature let event try { - event = stripe.webhooks.constructEvent(rawBody, signature, webhookSecret) + event = stripe.webhooks.constructEvent(body, signature, webhookSecret) } catch (err: any) { console.error(`โŒ Webhook signature verification failed:`) console.error(` Error: ${err.message}`) console.error(` Signature: ${signature?.substring(0, 50)}...`) - console.error(` Body length: ${rawBody.length}`) + console.error(` Body length: ${body.length}`) console.error(` Webhook secret set: ${!!webhookSecret}`) console.error(` Webhook secret prefix: ${webhookSecret?.substring(0, 10)}...`) @@ -187,7 +65,7 @@ export async function POST(request: NextRequest) { debug: { hasSignature: !!signature, hasSecret: !!webhookSecret, - bodyLength: rawBody.length, + bodyLength: body.length, secretPrefix: webhookSecret?.substring(0, 10) } }, { status: 400 }) diff --git a/auto-analyst-frontend/components/admin/DateRangePicker.tsx b/auto-analyst-frontend/components/admin/DateRangePicker.tsx deleted file mode 100644 index a429b5bc..00000000 --- a/auto-analyst-frontend/components/admin/DateRangePicker.tsx +++ /dev/null @@ -1,91 +0,0 @@ -import { useState } from 'react'; -import { format } from 'date-fns'; -import { Calendar as CalendarIcon } from 'lucide-react'; -import { DateRange } from 'react-day-picker'; -import { Button } from '@/components/ui/button'; -import { Calendar } from '@/components/ui/calendar'; -import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover'; -import { cn } from '@/lib/utils'; - -interface DateRangePickerProps { - dateRange: DateRange; - onDateRangeChange: (dateRange: DateRange) => void; - className?: string; - disabled?: boolean; -} - -export function DateRangePicker({ - dateRange, - onDateRangeChange, - className, - disabled = false -}: DateRangePickerProps) { - const [isOpen, setIsOpen] = useState(false); - - const handlePresetClick = (days: number) => { - const endDate = new Date(); - const startDate = new Date(); - startDate.setDate(endDate.getDate() - days); - - onDateRangeChange({ - from: startDate, - to: endDate - }); - setIsOpen(false); - }; - - return ( -
- - - - - -
- - - -
- { - if (range) { - onDateRangeChange(range); - } - }} - numberOfMonths={1} - /> -
- -
- - -
- ); -} \ No newline at end of file diff --git a/auto-analyst-frontend/components/admin/UsageTable.tsx b/auto-analyst-frontend/components/admin/UsageTable.tsx deleted file mode 100644 index b2d12363..00000000 --- a/auto-analyst-frontend/components/admin/UsageTable.tsx +++ /dev/null @@ -1,67 +0,0 @@ -import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table"; - -interface Column { - header: string; - accessor: string; - format?: 'currency' | 'number' | 'percentage' | 'date' | 'none'; -} - -interface UsageTableProps { - data: any[]; - columns: Column[]; - className?: string; -} - -export function UsageTable({ data, columns, className = '' }: UsageTableProps) { - const formatValue = (value: any, format?: string) => { - if (value === null || value === undefined) { - return 'N/A'; - } - - switch (format) { - case 'currency': - return `$${parseFloat(value).toFixed(2)}`; - case 'number': - return parseFloat(value).toLocaleString(); - case 'percentage': - return `${(parseFloat(value) * 100).toFixed(1)}%`; - case 'date': - return new Date(value).toLocaleDateString(); - default: - return value; - } - }; - - if (!data || data.length === 0) { - return ( -
- No data available -
- ); - } - - return ( -
- - - - {columns.map((column) => ( - {column.header} - ))} - - - - {data.map((row, rowIndex) => ( - - {columns.map((column) => ( - - {formatValue(row[column.accessor], column.format)} - - ))} - - ))} - -
-
- ); -} \ No newline at end of file diff --git a/auto-analyst-frontend/components/chat/ChatSettingsModal.tsx b/auto-analyst-frontend/components/chat/ChatSettingsModal.tsx deleted file mode 100644 index 2ec58ff5..00000000 --- a/auto-analyst-frontend/components/chat/ChatSettingsModal.tsx +++ /dev/null @@ -1,36 +0,0 @@ -"use client" - -import { useEffect } from 'react' -import { useModelSettings } from "@/lib/hooks/useModelSettings" -import SettingsPopup from './SettingsPopup' -import logger from '@/lib/utils/logger' - -interface ChatSettingsModalProps { - isOpen: boolean - onClose: () => void -} - -const ChatSettingsModal = ({ isOpen, onClose }: ChatSettingsModalProps) => { - const { modelSettings, fetchModelSettings } = useModelSettings() - - // Fetch latest settings when modal is opened - useEffect(() => { - if (isOpen) { - fetchModelSettings(); - } - }, [isOpen, fetchModelSettings]); - - return ( - { - // Refresh from server - fetchModelSettings() - }} - /> - ) -} - -export default ChatSettingsModal \ No newline at end of file diff --git a/auto-analyst-frontend/components/landing/LandingLayout.tsx b/auto-analyst-frontend/components/landing/LandingLayout.tsx deleted file mode 100644 index de7abc75..00000000 --- a/auto-analyst-frontend/components/landing/LandingLayout.tsx +++ /dev/null @@ -1,29 +0,0 @@ -import React, { ReactNode } from 'react'; -import Head from 'next/head'; -import Footer from './Footer'; - -interface LandingLayoutProps { - children: ReactNode; - title?: string; - description?: string; -} - -export default function LandingLayout({ - children, - title = 'Auto-Analyst - Transform Your Data Into Insights', - description = 'Harness the power of AI to analyze, predict, and optimize your business decisions' -}: LandingLayoutProps) { - return ( -
- - {title} - - - - - {children} - -
-
- ); -} \ No newline at end of file diff --git a/auto-analyst-frontend/components/landing/TestimoniaslSections.tsx b/auto-analyst-frontend/components/landing/TestimoniaslSections.tsx deleted file mode 100644 index 3dd05105..00000000 --- a/auto-analyst-frontend/components/landing/TestimoniaslSections.tsx +++ /dev/null @@ -1,44 +0,0 @@ -// TODO: Add testimonials section when we have some! - -// "use client" -// import { motion } from "framer-motion" -// import { testimonials } from "./FnT" - - -// export default function TestimonialsSection() { -// return ( -//
-//
-// -//

What Our Users Say

-//

Join thousands of satisfied customers

-// - -//
-// {testimonials.map((testimonial, index) => ( -// -//

"{testimonial.quote}"

-//
-//

{testimonial.author}

-//

{testimonial.role}

-//
-// -// ))} -//
-//
-//
-// ) -// } diff --git a/auto-analyst-frontend/docs/README.md b/auto-analyst-frontend/docs/README.md index 422a4273..9604330a 100644 --- a/auto-analyst-frontend/docs/README.md +++ b/auto-analyst-frontend/docs/README.md @@ -138,7 +138,7 @@ components/ ## ๐Ÿ—๏ธ Frontend Architecture ### **Tech Stack** -- **Next.js 14** - React framework with App Router +- **Next.js 13** - React framework with App Router - **TypeScript** - Full type safety - **Tailwind CSS** - Styling framework - **NextAuth.js** - Authentication diff --git a/auto-analyst-frontend/docs/architecture/overview.md b/auto-analyst-frontend/docs/architecture/overview.md index b1bbfd33..83b4d95f 100644 --- a/auto-analyst-frontend/docs/architecture/overview.md +++ b/auto-analyst-frontend/docs/architecture/overview.md @@ -1,6 +1,6 @@ # Frontend Architecture Overview -The Auto-Analyst frontend is built with **Next.js 14** using the App Router pattern, providing a modern, scalable architecture for an AI-powered analytics platform. +The Auto-Analyst frontend is built with **Next.js 13** using the App Router pattern, providing a modern, scalable architecture for an AI-powered analytics platform. ## ๐Ÿ—๏ธ Tech Stack @@ -18,7 +18,6 @@ The Auto-Analyst frontend is built with **Next.js 14** using the App Router patt ### **State Management** - **React Context** - Global state for auth, credits - **Zustand** - Client-side state store -- **SWR** - Data fetching and caching ### **Authentication** - **NextAuth.js** - Authentication framework @@ -214,7 +213,6 @@ eventSource.onmessage = (event) => { ### **Caching** - Redis for session data - Browser caching for static assets -- SWR for API response caching ### **Bundle Optimization** - Next.js built-in optimizations diff --git a/auto-analyst-frontend/docs/communication/api-integration.md b/auto-analyst-frontend/docs/communication/api-integration.md index 39638aae..1acbba7b 100644 --- a/auto-analyst-frontend/docs/communication/api-integration.md +++ b/auto-analyst-frontend/docs/communication/api-integration.md @@ -238,21 +238,6 @@ const userId = token?.sub; ## ๐Ÿ“ก Data Fetching Patterns -### **SWR for Client-side Fetching** -```typescript -import useSWR from 'swr'; - -const fetcher = (url: string) => fetch(url).then(res => res.json()); - -function UserCredits() { - const { data, error, isLoading } = useSWR('/api/user/credits', fetcher); - - if (error) return
Failed to load credits
; - if (isLoading) return
Loading...
; - - return
Credits: {data.remaining}
; -} -``` ### **React Query Alternative** ```typescript diff --git a/auto-analyst-frontend/docs/features/chat-system.md b/auto-analyst-frontend/docs/features/chat-system.md index 4280f311..973cf55d 100644 --- a/auto-analyst-frontend/docs/features/chat-system.md +++ b/auto-analyst-frontend/docs/features/chat-system.md @@ -481,55 +481,3 @@ const useChatScroll = (messages: ChatMessage[]) => { return { messagesEndRef, handleScroll } } ``` - -## ๐Ÿ”ง Performance Optimizations - -### **Message Virtualization** -For large chat histories, the system implements virtual scrolling: - -```typescript -const VirtualizedChatWindow = ({ messages }: { messages: ChatMessage[] }) => { - const { FixedSizeList: List } = require('react-window') - - const MessageItem = ({ index, style }: { index: number; style: any }) => ( -
- -
- ) - - return ( - - {MessageItem} - - ) -} -``` - -### **Debounced Input** -```typescript -const useDebouncedInput = (callback: (value: string) => void, delay: number) => { - const [inputValue, setInputValue] = useState('') - const debouncedValue = useDebounce(inputValue, delay) - - useEffect(() => { - if (debouncedValue) { - callback(debouncedValue) - } - }, [debouncedValue, callback]) - - return [inputValue, setInputValue] as const -} -``` - -### **Memoized Components** -```typescript -const MemoizedMessageContent = React.memo(MessageContent, (prevProps, nextProps) => { - return prevProps.message.id === nextProps.message.id && - prevProps.message.text === nextProps.message.text -}) -``` \ No newline at end of file diff --git a/auto-analyst-frontend/docs/state-management/stores.md b/auto-analyst-frontend/docs/state-management/stores.md index fa86cc2f..b2984f70 100644 --- a/auto-analyst-frontend/docs/state-management/stores.md +++ b/auto-analyst-frontend/docs/state-management/stores.md @@ -169,92 +169,6 @@ export function useCredits() { ### **2. Deep Analysis Provider** Manages deep analysis feature state. -```typescript -// lib/contexts/deep-analysis-context.tsx -interface DeepAnalysisContextType { - isAnalysisRunning: boolean - currentStep: string - progress: number - results: AnalysisResult[] - startAnalysis: (data: any) => Promise - stopAnalysis: () => void - clearResults: () => void -} - -export function DeepAnalysisProvider({ children }: { children: ReactNode }) { - const [isAnalysisRunning, setIsAnalysisRunning] = useState(false) - const [currentStep, setCurrentStep] = useState('') - const [progress, setProgress] = useState(0) - const [results, setResults] = useState([]) - - const startAnalysis = async (data: any) => { - setIsAnalysisRunning(true) - setProgress(0) - setCurrentStep('Initializing...') - - try { - const response = await fetch('/api/deep-analysis/start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(data) - }) - - // Handle streaming response for progress updates - const reader = response.body?.getReader() - const decoder = new TextDecoder() - - while (true) { - const { done, value } = await reader.read() - if (done) break - - const chunk = decoder.decode(value) - const lines = chunk.split('\n') - - for (const line of lines) { - if (line.startsWith('data: ')) { - const data = JSON.parse(line.slice(6)) - - if (data.step) setCurrentStep(data.step) - if (data.progress) setProgress(data.progress) - if (data.result) setResults(prev => [...prev, data.result]) - } - } - } - } catch (error) { - console.error('Analysis error:', error) - } finally { - setIsAnalysisRunning(false) - setCurrentStep('') - setProgress(100) - } - } - - const stopAnalysis = () => { - setIsAnalysisRunning(false) - setCurrentStep('') - } - - const clearResults = () => { - setResults([]) - setProgress(0) - } - - return ( - - {children} - - ) -} -``` - ## ๐Ÿ—„๏ธ Zustand Stores ### **1. Chat History Store** diff --git a/auto-analyst-frontend/docs/system/redis-schema.md b/auto-analyst-frontend/docs/system/redis-schema.md index 004a1890..8271cb73 100644 --- a/auto-analyst-frontend/docs/system/redis-schema.md +++ b/auto-analyst-frontend/docs/system/redis-schema.md @@ -88,7 +88,7 @@ await redis.hset(KEYS.USER_SUBSCRIPTION(userId), { plan: "pro", status: "active", interval: "month", - amount: "29.99", + amount: "15", stripeSubscriptionId: "sub_...", nextBilling: "2024-02-01" }) diff --git a/auto-analyst-frontend/docs/system/webhooks.md b/auto-analyst-frontend/docs/system/webhooks.md index b4224236..55a8418d 100644 --- a/auto-analyst-frontend/docs/system/webhooks.md +++ b/auto-analyst-frontend/docs/system/webhooks.md @@ -16,7 +16,7 @@ Auto-Analyst processes webhooks from Stripe to handle subscription and payment e // app/api/webhooks/route.ts export async function POST(request: Request) { const body = await request.text() - const sig = request.headers.get('stripe-signature') + const sig = headers().get('Stripe-Signature') let event: Stripe.Event diff --git a/auto-analyst-frontend/lib/README-Credits.md b/auto-analyst-frontend/lib/README-Credits.md deleted file mode 100644 index 6c787a74..00000000 --- a/auto-analyst-frontend/lib/README-Credits.md +++ /dev/null @@ -1,141 +0,0 @@ -# Centralized Credit Configuration - -This document explains the centralized credit allocation system implemented in `lib/credits-config.ts`. - -## Overview - -Previously, credit values were hardcoded throughout the application, making it difficult to maintain and update. The centralized configuration system consolidates all credit-related logic into a single file. - -## Configuration File - -### Location -`lib/credits-config.ts` - -### Key Components - -#### Plan Configuration -```typescript -export const PLAN_CREDITS: Record = { - 'Free': { - total: 100, - displayName: 'Free Plan', - type: 'FREE', - isUnlimited: false, - minimum: 0 - }, - 'Standard': { - total: 500, - displayName: 'Standard Plan', - type: 'STANDARD', - isUnlimited: false, - minimum: 0 - }, - 'Pro': { - total: 999999, - displayName: 'Pro Plan', - type: 'PRO', - isUnlimited: true, - minimum: 0 - } -} -``` - -#### Credit Thresholds -```typescript -export const CREDIT_THRESHOLDS: CreditThresholds = { - unlimitedThreshold: 99999, - defaultInitial: 100, - warningThreshold: 80 // Warn when user has used 80% of credits -} -``` - -## Utility Methods - -The `CreditConfig` class provides utility methods for: - -- **Plan Management**: `getCreditsForPlan()`, `getCreditsByType()` -- **Credit Validation**: `isUnlimitedPlan()`, `isUnlimitedTotal()` -- **Display Formatting**: `formatCreditTotal()`, `formatRemainingCredits()` -- **Usage Calculations**: `calculateUsagePercentage()`, `shouldWarnLowCredits()` - -## Usage Examples - -### Getting Credits for a Plan -```typescript -import { CreditConfig } from '@/lib/credits-config' - -// Get credits for a specific plan -const standardCredits = CreditConfig.getCreditsForPlan('Standard') -console.log(standardCredits.total) // 500 - -// Get credits by plan type -const proCredits = CreditConfig.getCreditsByType('PRO') -console.log(proCredits.total) // 999999 -``` - -### Formatting Credits for Display -```typescript -// Format credit totals -const totalDisplay = CreditConfig.formatCreditTotal(999999) // "Unlimited" -const totalDisplay2 = CreditConfig.formatCreditTotal(500) // "500" - -// Format remaining credits -const remaining = CreditConfig.formatRemainingCredits(50, 999999) // "Unlimited" -const remaining2 = CreditConfig.formatRemainingCredits(50, 500) // "450" -``` - -### Usage Percentage Calculations -```typescript -// Calculate usage percentage -const percentage = CreditConfig.calculateUsagePercentage(80, 100) // 80 -const percentageUnlimited = CreditConfig.calculateUsagePercentage(100, 999999) // 5 (small value for UI) - -// Check if user should be warned -const shouldWarn = CreditConfig.shouldWarnLowCredits(85, 100) // true (85% usage) -``` - -## Files Updated -git -The following files have been updated to use the centralized configuration: - -### Frontend -- `app/account/page.tsx` - Account page credit display -- `app/pricing/page.tsx` - Pricing page (if applicable) - -### Backend APIs -- `app/api/user/data/route.ts` - User data retrieval -- `app/api/update-credits/route.ts` - Credit updates after payment -- `app/api/webhooks/route.ts` - Stripe webhook handling -- `app/api/user/cancel-subscription/route.ts` - Subscription cancellation -- `app/api/initialize-credits/route.ts` - Credit initialization - -### Utilities -- `lib/redis.ts` - Redis utilities for credit management - -## Benefits - -1. **Single Source of Truth**: All credit values are defined in one place -2. **Easy Updates**: Change credit values by updating the config file -3. **Type Safety**: TypeScript interfaces ensure consistency -4. **Centralized Logic**: Credit formatting and validation logic is reusable -5. **Maintainable**: Easier to understand and modify credit behavior - -## Making Changes - -To update credit values: - -1. Edit `lib/credits-config.ts` -2. Update the `PLAN_CREDITS` object with new values -3. The changes will automatically apply across the entire application - -To add a new plan: - -1. Add the plan to the `PLAN_CREDITS` object -2. Update the `PlanName` type if needed -3. The new plan will be available throughout the application - -## Migration Notes - -- All hardcoded credit values have been replaced with centralized config calls -- The behavior remains the same, but is now centrally managed -- No database migration is required as this only affects application logic \ No newline at end of file diff --git a/auto-analyst-frontend/lib/server/redis-server.ts b/auto-analyst-frontend/lib/server/redis-server.ts deleted file mode 100644 index 26b6953b..00000000 --- a/auto-analyst-frontend/lib/server/redis-server.ts +++ /dev/null @@ -1,53 +0,0 @@ -import { Redis } from '@upstash/redis'; -import { KEYS } from '../redis'; -import { CreditConfig } from '../credits-config'; - -// Initialize Redis client for server-side operations only -const serverRedis = new Redis({ - url: process.env.UPSTASH_REDIS_REST_URL || '', - token: process.env.UPSTASH_REDIS_REST_TOKEN || '', -}); - -// Server-side only Redis operations -export const serverCreditUtils = { - async getRemainingCredits(userId: string): Promise { - try { - // Try getting from the hash first - const creditsHash = await serverRedis.hgetall(KEYS.USER_CREDITS(userId)); - - if (creditsHash && creditsHash.total) { - const total = parseInt(creditsHash.total as string); - const used = creditsHash.used ? parseInt(creditsHash.used as string) : 0; - return total - used; - } - - // No free credits anymore - users need to pay for trial - return 0; - } catch (error) { - console.error('Server Redis error fetching credits:', error); - return 0; // No free credits anymore - users need to pay for trial - } - } -}; - -// Test server-side Redis connection -export const testServerRedisConnection = async (): Promise<{ - success: boolean; - message: string; -}> => { - try { - await serverRedis.ping(); - return { - success: true, - message: 'Server Redis connection successful' - }; - } catch (error) { - console.error('Server Redis connection failed:', error); - return { - success: false, - message: error instanceof Error ? error.message : 'Unknown error' - }; - } -}; - -export default serverRedis; \ No newline at end of file diff --git a/auto-analyst-frontend/upstash-cron.js b/auto-analyst-frontend/upstash-cron.js deleted file mode 100644 index e69de29b..00000000 diff --git a/docs/README.md b/docs/README.md index 92be07b1..025d6e15 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,7 +8,7 @@ Auto-Analyst is an AI-powered analytics platform that enables users to analyze d ``` Auto-Analyst Platform Architecture -โ”œโ”€โ”€ Frontend (Next.js 14) +โ”œโ”€โ”€ Frontend โ”‚ โ”œโ”€โ”€ Chat Interface with AI Agents โ”‚ โ”œโ”€โ”€ Credit Management System โ”‚ โ”œโ”€โ”€ User Authentication & Authorization @@ -21,9 +21,8 @@ Auto-Analyst Platform Architecture โ”‚ โ”œโ”€โ”€ Database Integration (PostgreSQL/SQLite) โ”‚ โ””โ”€โ”€ Analytics & Usage Tracking โ””โ”€โ”€ Infrastructure - โ”œโ”€โ”€ Redis (Caching & Sessions) + โ”œโ”€โ”€ Redis (Credits and Subscriptions Tracking) โ”œโ”€โ”€ Stripe (Payments & Subscriptions) - โ”œโ”€โ”€ Cloud Storage (AWS S3/Vercel Blob) โ””โ”€โ”€ AI Models (OpenAI, Anthropic, Google, etc.) ``` @@ -161,15 +160,14 @@ cd auto-analyst-frontend npm run dev # Start development server npm run build # Build for production npm run lint # Run ESLint -npm run type-check # TypeScript checking ``` ### **Backend Development** ```bash cd auto-analyst-backend -uvicorn app:app --reload # Start with hot reload -python -m pytest # Run tests -python scripts/init_db.py # Initialize database +python -m app # Start with hot reload +python -m src.db.init_db # Initialize database +python -m scripts.populate_agent_templates # Fill DB with custom + default agents ``` ### **Full Stack Development** @@ -184,7 +182,7 @@ npm run dev:all # If configured in root package.json ## ๐Ÿ—๏ธ Architecture Decisions ### **Frontend Architecture** -- **Next.js 14**: App Router for modern React patterns +- **Next.js 13**: App Router for modern React patterns - **TypeScript**: Full type safety across the application - **Tailwind CSS**: Utility-first styling with custom components - **Zustand**: Lightweight state management for complex UI state @@ -200,7 +198,6 @@ npm run dev:all # If configured in root package.json ### **Data Architecture** - **PostgreSQL**: Primary database for production - **Redis**: Caching, session storage, and real-time data -- **S3/Blob Storage**: File uploads and static assets - **Real-time Updates**: WebSocket connections for live data ## ๐Ÿ”— Integration Points @@ -220,7 +217,7 @@ npm run dev:all # If configured in root package.json - **Vercel**: Frontend hosting and edge functions - **Upstash**: Managed Redis service - **HuggingFace**: AI model hosting and deployment -- **AWS**: Cloud infrastructure and storage +- **AWS**: Cloud infrastructure and storage (AWS RDS) ## ๐Ÿ“ˆ Monitoring & Analytics @@ -278,4 +275,4 @@ npm run dev:all # If configured in root package.json --- -This documentation serves as the central hub for understanding and working with the Auto-Analyst platform. For specific implementation details, refer to the frontend and backend documentation in their respective directories. \ No newline at end of file +This documentation serves as the central hub for understanding and working with the Auto-Analyst. For specific implementation details, refer to the frontend and backend documentation in their respective directories. \ No newline at end of file diff --git a/docs/backend.md b/docs/backend.md index 0510539c..53e14828 100644 --- a/docs/backend.md +++ b/docs/backend.md @@ -2,7 +2,7 @@ The **Auto-Analyst** backend provides a comprehensive API for data analysis, AI-powered insights, and real-time analytics. The API is organized into specialized route categories, each documented separately for better modularity: -1. **[Core Application Routes](auto-analyst-backend/docs/api/routes/core.md)** โ€“ Data management, session control, model configurations, and basic AI analysis. +1. **[Core Application Routes](auto-analyst-backend/docs/api/routes/session.md)** โ€“ Data management, session control, model configurations, and basic AI analysis. 2. **[Chat Management Routes](auto-analyst-backend/docs/api/routes/chats.md)** โ€“ Chat sessions, message handling, and user management. 3. **[Code Execution Routes](auto-analyst-backend/docs/api/routes/code.md)** โ€“ Python code execution, editing, fixing, and cleaning operations. 4. **[Deep Analysis Routes](auto-analyst-backend/docs/api/routes/deep_analysis.md)** โ€“ Advanced multi-agent analysis and comprehensive reporting. @@ -13,7 +13,7 @@ The **Auto-Analyst** backend provides a comprehensive API for data analysis, AI- ## **1. Core Application Routes** **Purpose**: Foundation for data management and basic AI analysis -**Documentation**: [auto-analyst-backend/docs/api/routes/core.md](auto-analyst-backend/docs/api/routes/core.md) +**Documentation**: [auto-analyst-backend/docs/api/routes/session.md](auto-analyst-backend/docs/api/routes/session.md) **Key Features**: - **Data Management**: Upload CSV/Excel files, preview datasets, reset sessions diff --git a/docs/frontend.md b/docs/frontend.md index cf29ae6c..5770d74a 100644 --- a/docs/frontend.md +++ b/docs/frontend.md @@ -1,10 +1,10 @@ # Auto-Analyst Frontend Overview ## 1. Frontend Architecture -The Auto-Analyst frontend is built using **Next.js 14** with the **App Router** pattern. The application follows a component-based architecture with clear separation of concerns between UI, business logic, and data management. +The Auto-Analyst frontend is built using **Next.js 13** with the **App Router** pattern. The application follows a component-based architecture with clear separation of concerns between UI, business logic, and data management. ### Tech Stack -- **Framework**: Next.js 14 with App Router +- **Framework**: Next.js 13 with App Router - **Language**: TypeScript for full type safety - **Styling**: Tailwind CSS with shadcn/ui components - **Authentication**: NextAuth.js with Google OAuth @@ -162,7 +162,6 @@ npm run type-check # TypeScript validation ### Caching Strategy - **Redis caching** for user sessions and data - **Browser caching** for static assets -- **SWR/React Query** for API response caching ---