refactor(memory): replace truncation with parallel chunked extraction

server/docs/API.md

@@ -6,8 +6,8 @@ The NeuraMemory-AI API is a RESTful interface for managing user memories and int
 
 The most up-to-date and interactive documentation (OpenAPI 3.0) is available directly through the server:
 
--   **Endpoint**: `/api-docs` (e.g., `http://localhost:3000/api-docs`)
--   **JSON Spec**: `/api-docs/spec.json`
+- **Endpoint**: `/api-docs` (e.g., `http://localhost:3000/api-docs`)
+- **JSON Spec**: `/api-docs/spec.json`
 
 Use the Swagger UI to test endpoints directly from your browser.
 
@@ -18,24 +18,28 @@ Use the Swagger UI to test endpoints directly from your browser.
 Most endpoints require authentication. We support two methods:
 
 ### 1. JWT (Web Application)
+
 Used by the React frontend.
--   **Method**: `Authorization: Bearer <token>`
--   **Obtain**: Call `/api/v1/login`.
+
+- **Method**: `Authorization: Bearer <token>`
+- **Obtain**: Call `/api/v1/login`.
 
 ### 2. API Key (MCP & Integrations)
+
 Used by external tools like the Claude Desktop MCP.
--   **Method**: `x-api-key: <key>` header or `?apiKey=<key>` query parameter (SSE only).
--   **Obtain**: Generate in the UI or via `/api/v1/api-key`.
+
+- **Method**: `x-api-key: <key>` header or `?apiKey=<key>` query parameter (SSE only).
+- **Obtain**: Generate in the UI or via `/api/v1/api-key`.
 
 ---
 
 ## 🚦 Rate Limiting
 
 To ensure stability, we apply rate limits based on the user's ID or IP.
 
--   **Auth Endpoints**: Strict limits to prevent brute-force attacks.
--   **Memory Endpoints**: Moderate limits to prevent excessive LLM/Embedding usage.
--   **Chat Endpoints**: Configurable window (e.g., 30 requests/minute in production).
+- **Auth Endpoints**: Strict limits to prevent brute-force attacks.
+- **Memory Endpoints**: Moderate limits to prevent excessive LLM/Embedding usage.
+- **Chat Endpoints**: Configurable window (e.g., 30 requests/minute in production).
 
 When a limit is reached, the API returns a `429 Too Many Requests` status code.
 
@@ -54,20 +58,20 @@ All errors return a consistent JSON body:
 
 ### Common Status Codes
 
-| Code | Name | Description |
-| :--- | :--- | :--- |
-| `400` | Bad Request | Validation failed (Zod error) |
-| `401` | Unauthorized | Missing or invalid authentication |
-| `403` | Forbidden | Insufficient permissions |
-| `409` | Conflict | Resource already exists (e.g., email) |
+| Code  | Name                 | Description                             |
+| :---- | :------------------- | :-------------------------------------- |
+| `400` | Bad Request          | Validation failed (Zod error)           |
+| `401` | Unauthorized         | Missing or invalid authentication       |
+| `403` | Forbidden            | Insufficient permissions                |
+| `409` | Conflict             | Resource already exists (e.g., email)   |
 | `422` | Unprocessable Entity | Logic error (e.g., failed to fetch URL) |
-| `429` | Too Many Requests | Rate limit exceeded |
-| `500` | Internal Error | Unexpected server-side failure |
+| `429` | Too Many Requests    | Rate limit exceeded                     |
+| `500` | Internal Error       | Unexpected server-side failure          |
 
 ---
 
 ## 🛠 Standard Headers
 
--   `Content-Type: application/json`
--   `Accept: application/json`
--   `x-csrf-token`: Required for state-changing requests (POST/PATCH/DELETE) in the web app to prevent cross-site request forgery.
+- `Content-Type: application/json`
+- `Accept: application/json`
+- `x-csrf-token`: Required for state-changing requests (POST/PATCH/DELETE) in the web app to prevent cross-site request forgery.

server/docs/ARCHITECTURE.md

@@ -7,24 +7,24 @@ NeuraMemory-AI follows a clean, layered architecture designed for scalability, m
 The backend is structured into four primary layers:
 
 1.  **Routes Layer** (`/src/routes`):
-    *   Defines HTTP endpoints.
-    *   Applies middleware (auth, rate limiting, file uploads).
-    *   Delegates requests to Controllers.
+    - Defines HTTP endpoints.
+    - Applies middleware (auth, rate limiting, file uploads).
+    - Delegates requests to Controllers.
 
 2.  **Controller Layer** (`/src/controllers`):
-    *   Handles HTTP-specific logic (parsing params, sending responses).
-    *   Validates request bodies using Zod schemas.
-    *   Calls one or more Services to fulfill the request.
+    - Handles HTTP-specific logic (parsing params, sending responses).
+    - Validates request bodies using Zod schemas.
+    - Calls one or more Services to fulfill the request.
 
 3.  **Service Layer** (`/src/services`):
-    *   Contains the core business logic.
-    *   Orchestrates complex flows (e.g., fetching a link, extracting text, calling LLM, generating embeddings).
-    *   Agnotic of the HTTP layer.
+    - Contains the core business logic.
+    - Orchestrates complex flows (e.g., fetching a link, extracting text, calling LLM, generating embeddings).
+    - Agnotic of the HTTP layer.
 
 4.  **Repository Layer** (`/src/repositories`):
-    *   Handles all data persistence and retrieval.
-    *   Interacts directly with PostgreSQL (metadata) and Qdrant (vector embeddings).
-    *   Provides a clean API for the Services to store and find data.
+    - Handles all data persistence and retrieval.
+    - Interacts directly with PostgreSQL (metadata) and Qdrant (vector embeddings).
+    - Provides a clean API for the Services to store and find data.
 
 ---
 
@@ -37,8 +37,8 @@ When a user submits a piece of information (text, link, or document), the follow
 3.  **Memory Synthesis**: The LLM (via OpenRouter) identifies "Semantic Facts" (facts that don't change often) and "Episodic Bubbles" (event-based memories).
 4.  **Embedding**: Each extracted memory is converted into a vector representation using an embedding model.
 5.  **Storage**:
-    *   **PostgreSQL**: Stores user metadata and API keys.
-    *   **Qdrant**: Stores the vector embeddings and memory text for semantic search.
+    - **PostgreSQL**: Stores user metadata and API keys.
+    - **Qdrant**: Stores the vector embeddings and memory text for semantic search.
 
 ---
 
@@ -55,7 +55,7 @@ When a user chats with their "Memory Hub":
 
 ## 🛠 Key Infrastructure
 
--   **PostgreSQL**: Reliable relational storage for structured data (Users, Accounts, Sessions).
--   **Qdrant**: High-performance vector database for semantic search and high-dimensional data retrieval.
--   **OpenRouter**: Unified gateway for accessing state-of-the-art LLMs (Claude, GPT, Gemini) for extraction and conversation.
--   **Firecrawl**: (Optional/Planned) For robust web scraping and link extraction.
+- **PostgreSQL**: Reliable relational storage for structured data (Users, Accounts, Sessions).
+- **Qdrant**: High-performance vector database for semantic search and high-dimensional data retrieval.
+- **OpenRouter**: Unified gateway for accessing state-of-the-art LLMs (Claude, GPT, Gemini) for extraction and conversation.
+- **Firecrawl**: (Optional/Planned) For robust web scraping and link extraction.

server/docs/BEST_PRACTICES.md

@@ -4,18 +4,18 @@ This document outlines the conventions and patterns used in the NeuraMemory-AI s
 
 ## 🛠 Technology & Language
 
--   **TypeScript Everywhere**: Use strict typing. Avoid `any` at all costs.
--   **ES Modules (ESM)**: We use `"type": "module"` in `package.json`. All imports must include the `.js` extension (e.g., `import { user } from './user.js'`).
--   **Node.js 24+**: Leverage modern Node.js features like built-in `test` runner (if applicable) and top-level await.
+- **TypeScript Everywhere**: Use strict typing. Avoid `any` at all costs.
+- **ES Modules (ESM)**: We use `"type": "module"` in `package.json`. All imports must include the `.js` extension (e.g., `import { user } from './user.js'`).
+- **Node.js 24+**: Leverage modern Node.js features like built-in `test` runner (if applicable) and top-level await.
 
 ---
 
 ## 🚦 Error Handling
 
-Use the centralized `AppError` class located in `src/utils/AppError.ts`. 
+Use the centralized `AppError` class located in `src/utils/AppError.ts`.
 
--   **Pattern**: Do not use `try/catch` in controllers for simple error wrapping. Throw an `AppError` or let the `errorHandler` middleware catch async errors.
--   **Status Codes**: Use appropriate HTTP status codes (400 for validation, 404 for missing resources, etc.).
+- **Pattern**: Do not use `try/catch` in controllers for simple error wrapping. Throw an `AppError` or let the `errorHandler` middleware catch async errors.
+- **Status Codes**: Use appropriate HTTP status codes (400 for validation, 404 for missing resources, etc.).
 
 ```typescript
 // Example
@@ -30,8 +30,8 @@ if (!user) {
 
 Every endpoint that accepts input (body, query, params) **must** validate it using [Zod](https://zod.dev/).
 
--   Define schemas in `src/validations/`.
--   Use the `parse` method to ensure type safety and early failure.
+- Define schemas in `src/validations/`.
+- Use the `parse` method to ensure type safety and early failure.
 
 ---
 
@@ -46,14 +46,14 @@ Every endpoint that accepts input (body, query, params) **must** validate it usi
 
 ## 🔒 Security
 
--   **JWT for Web**: Use cookies with `httpOnly: true` and `secure: true` in production.
--   **CSRF Protection**: All mutation requests (POST/PATCH/DELETE) require a CSRF token.
--   **Sanitization**: Never trust user input. Use parameterized queries (handled by our repository layer) and sanitize HTML content when extracting from links.
+- **JWT for Web**: Use cookies with `httpOnly: true` and `secure: true` in production.
+- **CSRF Protection**: All mutation requests (POST/PATCH/DELETE) require a CSRF token.
+- **Sanitization**: Never trust user input. Use parameterized queries (handled by our repository layer) and sanitize HTML content when extracting from links.
 
 ---
 
 ## 🧪 Testing
 
--   **Unit Tests**: Use `vitest` for testing utilities and services.
--   **API Tests**: Use the `test.sh` script to run end-to-end integration tests using `curl` and `jq`.
--   **Coverage**: Aim for high coverage in critical paths (auth, memory extraction).
+- **Unit Tests**: Use `vitest` for testing utilities and services.
+- **API Tests**: Use the `test.sh` script to run end-to-end integration tests using `curl` and `jq`.
+- **Coverage**: Aim for high coverage in critical paths (auth, memory extraction).

server/src/config/env.ts

@@ -8,58 +8,57 @@ const envSchema = z.object({
   NODE_ENV: z
     .enum(['development', 'production', 'test'])
     .default('development'),
-  
+
   // Database
   DATABASE_URL: z.string().url(),
   QDRANT_URL: z.string().url(),
   QDRANT_API_KEY: z.string().optional(),
-  
+
   // LLM / AI
   OPENROUTER_BASE_URL: z.string().url().default('https://openrouter.ai/api/v1'),
   OPENROUTER_API_KEY: z.string().min(1, 'OpenRouter API Key is required'),
   OPENROUTER_REFERER: z.string().url().optional(),
   OPENROUTER_TITLE: z.string().optional(),
   CHAT_MODEL: z.string().default('google/gemini-2.0-flash-001'),
-  EMBEDDING_MODEL: z.string().default('google/gemini-2.0-flash-001'),
-
-  // Scrapers / Ingestion
+
+  // Memory conflict / retrieval tuning
+  SIMILARITY_THRESHOLD: z.coerce.number().min(0).max(1).default(0.78),
+  DUPLICATE_THRESHOLD: z.coerce.number().min(0).max(1).default(0.92),
+  CONFLICT_CONFIDENCE_THRESHOLD: z.coerce.number().min(0).max(1).default(0.7),
+  RETRIEVAL_DEDUP_THRESHOLD: z.coerce.number().min(0).max(1).default(0.95),
+  CONFLICT_STRATEGY: z
+    .enum(['recency', 'confidence', 'merge', 'flag'])
+    .default('recency'),
+
+  JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 characters'),
+  JWT_EXPIRES_IN: z
+    .string()
+    .default('7d')
+    .transform((val) => val.split('#')[0]!.trim()),
+
+  // Content extraction
   FIRECRAWL_API_KEY: z.string().optional(),
-  CRAWL4AI_API_URL: z.string().url().default('http://localhost:11235'),
+  CRAWL4AI_API_URL: z.string().url().default('http://crawl4ai:11235'),
   UNSTRUCTURED_API_URL: z
     .string()
     .url()
     .default('https://platform.unstructuredapp.io/api/v1'),
   UNSTRUCTURED_API_KEY: z.string().optional(),
   UNSTRUCTURED_TIMEOUT_MS: z.string().optional(),
-  
+
   // OCR
   OCR_ENABLE_LOCAL_FALLBACK: z.string().default('true'),
   OCR_TESSERACT_LANG: z.string().default('eng'),
   OCR_FORCE: z.string().default('false'),
-
-  // Security
-  JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 characters'),
-  JWT_EXPIRES_IN: z
+  ALLOWED_ORIGINS: z
     .string()
-    .default('7d')
-    .transform((val) => val.split('#')[0]!.trim()),
-  ADMIN_API_KEY: z.string().min(1, 'ADMIN_API_KEY is required for secure operations'),
-
-  // App Logic
-  ALLOWED_ORIGINS: z.string().default('http://localhost:5173'),
-  SIMILARITY_THRESHOLD: z.coerce.number().default(0.82),
-  DUPLICATE_THRESHOLD: z.coerce.number().default(0.95),
-  CONFLICT_CONFIDENCE_THRESHOLD: z.coerce.number().default(0.75),
-  RETRIEVAL_DEDUP_THRESHOLD: z.coerce.number().default(0.88),
-  CONFLICT_STRATEGY: z
-    .enum(['recency', 'confidence', 'merge', 'flag'])
-    .default('recency'),
+    .default('http://localhost:5173,http://localhost:5174'),
 });
 
 const _env = envSchema.safeParse(process.env);
 
 if (!_env.success) {
-  const missing = _env.error.errors.map(e => e.path.join('.')).join(', ');
+  const missing = _env.error.errors.map((e) => e.path.join('.')).join(', ');
   console.error(`❌ Invalid or missing environment variables: ${missing}`);
   process.exit(1);
 }

server/src/controllers/auth.controller.ts

@@ -122,7 +122,8 @@ export async function logoutController(
   try {
     const userId = req.user?.userId;
     if (userId) {
-      const { revokeSessionsService } = await import('../services/auth.service.js');
+      const { revokeSessionsService } =
+        await import('../services/auth.service.js');
       await revokeSessionsService(userId);
     }
 
@@ -131,7 +132,10 @@ export async function logoutController(
       secure: env.NODE_ENV === 'production',
       sameSite: env.NODE_ENV === 'production' ? 'none' : 'lax',
     });
-    res.status(200).json({ success: true, message: 'Logged out successfully. All sessions revoked.' });
+    res.status(200).json({
+      success: true,
+      message: 'Logged out successfully. All sessions revoked.',
+    });
   } catch (err) {
     next(err);
   }

server/src/index.ts

@@ -36,15 +36,21 @@ app.use(
   cors({
     origin: (origin, callback) => {
       // Allow if no origin (e.g., server to server, Postman) or if origin is in whitelist
-      if (!origin || allowedOrigins.some(o => origin.startsWith(o))) {
+      if (!origin || allowedOrigins.some((o) => origin.startsWith(o))) {
         return callback(null, true);
       }
       logger.warn(`[CORS] Rejected Origin: ${origin}`);
       callback(new Error(`CORS: origin ${origin} not allowed`));
     },
     credentials: true,
     methods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'OPTIONS'],
-    allowedHeaders: ['Content-Type', 'Authorization', 'x-api-key', 'mcp-session-id', 'x-csrf-token'],
+    allowedHeaders: [
+      'Content-Type',
+      'Authorization',
+      'x-api-key',
+      'mcp-session-id',
+      'x-csrf-token',
+    ],
   }),
 );
 
@@ -101,7 +107,7 @@ async function shutdown(signal: string): Promise<void> {
     logger.info('[Shutdown] PostgreSQL pool closed.');
     closeQdrantClient();
     logger.info('[Shutdown] Qdrant client closed.');
-    
+
     process.exit(0);
   } catch (err) {
     logger.error('[Shutdown] Error during shutdown:', err);
@@ -112,7 +118,9 @@ async function shutdown(signal: string): Promise<void> {
 function registerProcessHandlers(): void {
   process.on('SIGINT', () => void shutdown('SIGINT'));
   process.on('SIGTERM', () => void shutdown('SIGTERM'));
-  process.on('unhandledRejection', (reason) => logger.error('[Process] Unhandled rejection:', reason));
+  process.on('unhandledRejection', (reason) =>
+    logger.error('[Process] Unhandled rejection:', reason),
+  );
   process.on('uncaughtException', (err) => {
     logger.error('[Process] Uncaught exception:', err);
     void shutdown('uncaughtException');
@@ -124,7 +132,9 @@ async function main(): Promise<void> {
 
   const port = Number(env.PORT);
   server = app.listen(port, () => {
-    logger.info(`[Startup] Server is listening on port ${port}. Starting dependency initialization...`);
+    logger.info(
+      `[Startup] Server is listening on port ${port}. Starting dependency initialization...`,
+    );
 
     // Run dependency verification in the background to avoid blocking the Startup Probe
     (async () => {
@@ -133,7 +143,9 @@ async function main(): Promise<void> {
         const { checkConnectivity } = await import('./lib/postgres.js');
         const isDbConnected = await checkConnectivity();
         if (!isDbConnected) {
-          throw new Error('Could not connect to PostgreSQL. Ensure the local service is running on port 5432.');
+          throw new Error(
+            'Could not connect to PostgreSQL. Ensure the local service is running on port 5432.',
+          );
         }
 
         await ensureDatabaseSchema();
@@ -145,7 +157,7 @@ async function main(): Promise<void> {
         logger.info('🚀 NeuraMemory-AI is fully operational.');
       } catch (err) {
         logger.error('[Startup] Initialization failed:', err);
-        // We don't exit in production here, allowing the health check to report the failure 
+        // We don't exit in production here, allowing the health check to report the failure
         // and standard retries to happen while keeping the process alive for debug/logs.
       }
     })();
@@ -154,9 +166,10 @@ async function main(): Promise<void> {
 
 registerProcessHandlers();
 
-const isMain = import.meta.url === `file://${process.argv[1]}` || 
-               process.argv[1]?.endsWith('index.ts') || 
-               process.argv[1]?.endsWith('index.js');
+const isMain =
+  import.meta.url === `file://${process.argv[1]}` ||
+  process.argv[1]?.endsWith('index.ts') ||
+  process.argv[1]?.endsWith('index.js');
 
 if (isMain) {
   main().catch((err) => {