Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintrc-auto-import.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"Chat": true,
"Dashboard": true,
"MainLayout": true,
"Pomodoro": true,
"Settings": true,
"createRef": true,
"forwardRef": true,
Expand Down
74 changes: 66 additions & 8 deletions .project.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,39 @@ const activities = await getActivities({ limit: 50 })

Always use auto-generated functions from `@/lib/client/apiClient` for type safety and automatic camelCase↔snake_case conversion.

### 2. Always Define Structured Request/Response Models
### 2. Always Use Relative Import for `api_handler`

**CRITICAL: Handler modules MUST use relative import to avoid circular import issues.**

```python
# ✅ CORRECT: Use relative import
from . import api_handler

# ❌ WRONG: Causes circular import
from handlers import api_handler
from backend.handlers import api_handler
```

**Why:** The `handlers/__init__.py` imports all handler modules at the bottom. Using absolute import causes Python to reload the package while it's still initializing, resulting in `api_handler` being undefined or import failures.

### 3. Always Define Structured Request/Response Models

**CRITICAL RULE:** All handlers must use Pydantic models inheriting from `backend.models.base.BaseModel`.

**NEVER use `Dict[str, Any]` as return type** - this prevents TypeScript type generation for the frontend.

```python
from backend.handlers import api_handler
from backend.models.base import BaseModel, TimedOperationResponse
from datetime import datetime
from typing import Optional

from core.logger import get_logger
from models.base import BaseModel
from models.responses import TimedOperationResponse

# ✅ CORRECT: Use relative import
from . import api_handler

logger = get_logger(__name__)

# ❌ WRONG - Dict prevents TypeScript type generation
@api_handler()
Expand Down Expand Up @@ -154,6 +178,15 @@ When frontend needs data:

**Data Flow:** RawRecords (60s memory) → Events (LLM) → Activities (10min aggregation) → Tasks (AI-generated)

**Pomodoro Mode:**

- **Core Principle**: Pomodoro mode ONLY controls whether perception layer (keyboard/mouse/screenshots) is running
- **Idle Mode (Default)**: Perception layer is stopped, no data captured
- **Active Mode (Pomodoro)**: Perception layer is running, captures user activity
- **No Configuration**: Pomodoro has NO system configuration parameters (e.g., default duration, behavior settings)
- **User-Controlled**: Duration is specified per session when starting, not a global config
- **Behavior Unchanged**: Capture behavior (smart capture, deduplication, etc.) follows normal settings and is NOT modified by Pomodoro mode

## Development Commands

```bash
Expand Down Expand Up @@ -182,23 +215,48 @@ pnpm sign-macos # Code signing (after bundle)

### Adding API Handler

**CRITICAL: Always use relative import `from . import api_handler` in handler modules to avoid circular import issues.**

```python
# 1. backend/handlers/my_feature.py
from backend.handlers import api_handler
from backend.models.base import BaseModel
from datetime import datetime

from core.coordinator import get_coordinator
from core.logger import get_logger
from models.base import BaseModel
from models.responses import TimedOperationResponse

# ✅ CORRECT: Use relative import to avoid circular imports
from . import api_handler

logger = get_logger(__name__)


class MyRequest(BaseModel):
user_input: str


@api_handler(body=MyRequest, method="POST", path="/endpoint", tags=["module"])
async def my_handler(body: MyRequest) -> dict:
return {"data": body.user_input}
async def my_handler(body: MyRequest) -> TimedOperationResponse:
return TimedOperationResponse(
success=True,
message="Operation completed",
timestamp=datetime.now().isoformat()
)

# 2. Import in backend/handlers/__init__.py
# 3. Run: pnpm setup-backend
# 3. Run: pnpm tauri:dev:gen-ts (to regenerate TypeScript bindings)
# 4. Use: import { myHandler } from '@/lib/client/apiClient'
```

**Why relative import is required:**

- `from . import api_handler` ✅ Correct - imports from current package (`handlers`)
- `from handlers import api_handler` ❌ Wrong - causes circular import because `handlers/__init__.py` is importing your module
- `from backend.handlers import api_handler` ❌ Wrong - same circular import issue

The `handlers/__init__.py` file imports all handler modules at the bottom (line 207-218). If your handler uses absolute import, Python will try to reload the `handlers` package while it's still being initialized, causing import failures.

### Adding i18n

```typescript
Expand Down
55 changes: 47 additions & 8 deletions backend/agents/action_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,36 @@ async def extract_and_save_actions(
try:
logger.debug(f"ActionAgent: Processing {len(records)} records")

# Pre-persist all screenshots to prevent cache eviction during LLM processing
screenshot_records = [
r for r in records if r.type == RecordType.SCREENSHOT_RECORD
]
screenshot_hashes = [
r.data.get("hash")
for r in screenshot_records
if r.data and r.data.get("hash")
]

if screenshot_hashes:
logger.debug(
f"ActionAgent: Pre-persisting {len(screenshot_hashes)} screenshots "
f"before LLM call to prevent cache eviction"
)
persist_results = self.image_manager.persist_images_batch(screenshot_hashes)

# Log pre-persistence results
success_count = sum(1 for success in persist_results.values() if success)
if success_count < len(screenshot_hashes):
logger.warning(
f"ActionAgent: Pre-persistence incomplete: "
f"{success_count}/{len(screenshot_hashes)} images persisted. "
f"Some images may already be evicted from cache."
)
else:
logger.debug(
f"ActionAgent: Successfully pre-persisted all {len(screenshot_hashes)} screenshots"
)

# Step 1: Extract actions using LLM
actions = await self._extract_actions(
records, input_usage_hint, keyboard_records, mouse_records, enable_supervisor
Expand All @@ -102,10 +132,7 @@ async def extract_and_save_actions(
return 0

# Step 2: Validate and resolve screenshot hashes
screenshot_records = [
r for r in records if r.type == RecordType.SCREENSHOT_RECORD
]

# (screenshot_records already created above for pre-persistence)
resolved_actions: List[Dict[str, Any]] = []
for action_data in actions:
action_hashes = self._resolve_action_screenshot_hashes(
Expand Down Expand Up @@ -455,12 +482,24 @@ def _persist_action_screenshots(self, screenshot_hashes: list[str]) -> None:

results = self.image_manager.persist_images_batch(screenshot_hashes)

# Log warnings for failed persists
# Enhanced logging for failed persists
failed = [h for h, success in results.items() if not success]
if failed:
logger.warning(
f"Failed to persist {len(failed)} screenshots (likely evicted from memory): "
f"{[h[:8] for h in failed]}"
logger.error(
f"ActionAgent: Image persistence FAILURE: {len(failed)}/{len(screenshot_hashes)} images lost. "
f"Action will be saved with broken image references. "
f"\nFailed hashes: {[h[:8] + '...' for h in failed[:5]]}"
f"{' (and ' + str(len(failed) - 5) + ' more)' if len(failed) > 5 else ''}"
f"\nRoot cause: Images evicted from memory cache before persistence."
f"\nRecommendations:"
f"\n 1. Increase memory_ttl in config.toml (current: {self.image_manager.memory_ttl}s, recommended: ≥180s)"
f"\n 2. Run GET /image/persistence-health to check system health"
f"\n 3. Run POST /image/cleanup-broken-actions to fix existing issues"
f"\n 4. Consider increasing memory_cache_size (current: {self.image_manager.memory_cache_size}, recommended: ≥1000)"
)
else:
logger.debug(
f"ActionAgent: Successfully persisted all {len(screenshot_hashes)} screenshots"
)

except Exception as e:
Expand Down
7 changes: 4 additions & 3 deletions backend/config/config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ enable_phash = true

# Memory-first storage configuration
enable_memory_first = true # Master switch
memory_ttl_multiplier = 2.5 # TTL = processing_interval * multiplier
memory_ttl_multiplier = 5.0 # TTL = processing_interval * multiplier (increased for better persistence)
memory_ttl_min = 60 # Minimum TTL (seconds)
memory_ttl_max = 120 # Maximum TTL (seconds)
memory_ttl_max = 300 # Maximum TTL (seconds) (increased from 120 to prevent eviction during LLM processing)

# Screenshot configuration
[screenshot]
Expand Down Expand Up @@ -141,7 +141,8 @@ crop_threshold = 30
# Memory cache size (images)
# Description: Cache recent image base64 data in memory
# Recommendation: 200-500 (memory usage ~100-250MB)
memory_cache_size = 500
# Increased to 1000 for better persistence reliability (memory usage ~500-1000MB)
memory_cache_size = 1000

# ========== Optimization Effect Estimation ==========
# Based on default configuration (aggressive + hybrid), with 20 original screenshots as example:
Expand Down
122 changes: 120 additions & 2 deletions backend/core/coordinator.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ def __init__(self, config: Dict[str, Any]):
self.knowledge_agent = None
self.diary_agent = None
self.cleanup_agent = None
self.pomodoro_manager = None

# Pomodoro mode state
self.pomodoro_mode = False
self.current_pomodoro_session_id: Optional[str] = None

# Running state
self.is_running = False
Expand Down Expand Up @@ -315,6 +320,11 @@ def _init_managers(self):
),
)

if self.pomodoro_manager is None:
from core.pomodoro_manager import PomodoroManager

self.pomodoro_manager = PomodoroManager(self)

# Link agents
if self.processing_pipeline:
# Link action_agent to pipeline for action extraction
Expand Down Expand Up @@ -401,13 +411,15 @@ async def start(self) -> None:
raise Exception("Cleanup agent initialization failed")

# Start all components in parallel (they are independent)
# NOTE: Perception manager is NOT started by default - it will be started
# when a Pomodoro session begins (Active mode strategy)
logger.debug(
"Starting perception manager, processing pipeline, agents in parallel..."
"Starting processing pipeline and agents (perception will start with Pomodoro)..."
)
start_time = datetime.now()

await asyncio.gather(
self.perception_manager.start(),
# self.perception_manager.start(), # Disabled: starts with Pomodoro
self.processing_pipeline.start(),
self.event_agent.start(),
self.session_agent.start(),
Expand All @@ -420,6 +432,12 @@ async def start(self) -> None:
f"All components started (took {elapsed:.2f}s)"
)

# Check for orphaned Pomodoro sessions from previous run
if self.pomodoro_manager:
orphaned_count = await self.pomodoro_manager.check_orphaned_sessions()
if orphaned_count > 0:
logger.info(f"✓ Recovered {orphaned_count} orphaned Pomodoro session(s)")

# Start scheduled processing loop
self.is_running = True
self._set_state(mode="running", error=None)
Expand Down Expand Up @@ -663,6 +681,106 @@ def get_stats(self) -> Dict[str, Any]:
return {"error": str(e)}


async def enter_pomodoro_mode(self, session_id: str) -> None:
"""
Enter Pomodoro mode - start perception and disable continuous processing

Changes:
1. Start perception manager (if not already running)
2. Stop processing_loop (cancel task)
3. Set pomodoro_mode = True
4. Set current_pomodoro_session_id
5. Perception captures and tags records
6. Records are saved to DB instead of processed

Args:
session_id: Pomodoro session identifier
"""
logger.info(f"→ Entering Pomodoro mode: {session_id}")

self.pomodoro_mode = True
self.current_pomodoro_session_id = session_id

# Start perception manager if not already running
if self.perception_manager and not self.perception_manager.is_running:
try:
logger.info("Starting perception manager for Pomodoro mode...")
await self.perception_manager.start()
logger.info("✓ Perception manager started")
except Exception as e:
logger.error(f"Failed to start perception manager: {e}", exc_info=True)
raise
elif not self.perception_manager:
logger.error("Perception manager is None, cannot start")
else:
logger.debug("Perception manager already running")

# Keep processing loop running - do NOT cancel it
# This allows Actions (30s) and Events (10min) to continue normally

# Pause only SessionAgent (activity generation deferred)
try:
if self.session_agent:
self.session_agent.pause()
logger.debug("✓ SessionAgent paused (activity generation deferred)")
except Exception as e:
logger.error(f"Failed to pause SessionAgent: {e}")

# Notify perception manager of Pomodoro mode (for tagging records)
if self.perception_manager:
self.perception_manager.set_pomodoro_session(session_id)

logger.info(
"✓ Pomodoro mode active - normal processing continues, "
"activity generation paused until session ends"
)

async def exit_pomodoro_mode(self) -> None:
"""
Exit Pomodoro mode - stop perception and trigger activity generation

When Pomodoro ends:
- Stop perception manager
- Resume SessionAgent
- Trigger immediate activity aggregation for accumulated Events
"""
logger.info("→ Exiting Pomodoro mode")

self.pomodoro_mode = False
session_id = self.current_pomodoro_session_id
self.current_pomodoro_session_id = None

# Stop perception manager
if self.perception_manager and self.perception_manager.is_running:
try:
logger.debug("Stopping perception manager...")
await self.perception_manager.stop()
logger.debug("✓ Perception manager stopped")
except Exception as e:
logger.error(f"Failed to stop perception manager: {e}")

# Processing loop is still running - no need to resume

# Resume SessionAgent and trigger immediate activity aggregation
try:
if self.session_agent:
self.session_agent.resume()
logger.debug("✓ SessionAgent resumed")

# Trigger immediate activity aggregation for Pomodoro session
logger.info("→ Triggering activity aggregation for Pomodoro session...")
await self.session_agent._aggregate_sessions()
logger.info("✓ Activity aggregation complete for Pomodoro session")
except Exception as e:
logger.error(f"Failed to resume SessionAgent or aggregate activities: {e}")

# Notify perception manager to exit Pomodoro mode
if self.perception_manager:
self.perception_manager.clear_pomodoro_session()

logger.info(f"✓ Idle mode resumed - perception stopped (exited session: {session_id})")


def get_coordinator() -> PipelineCoordinator:
"""Get global coordinator singleton"""
global _coordinator
Expand Down
Loading