Fix N+1 query issues in get_linked_todos and get_linked_events functions

[Copilot]

This PR eliminates N+1 query performance issues in the get_linked_todos() and get_linked_events() functions in next_crm/api/activities.py.

Problem

The original implementation executed database queries inside loops:

• get_linked_todos(): Queried Event details and Event Participants separately for each ToDo with a linked event
• get_linked_events(): Queried Event Participants for each event
• This created O(n) query patterns that scaled poorly with the number of records

Solution Implemented

get_linked_todos() function

• Batch fetch all unique linked events in a single query using ["in", event_names] filter
• Batch fetch all event participants in a single query
• Create lookup maps (events_map and participants_map) for O(1) access during data application
• Use sets to avoid querying duplicate event names

get_linked_events() function

• Batch fetch all event participants for all events in a single query using ["in", event_names] filter
• Create participants_map for O(1) access during data application
• Use sets to avoid querying duplicate event names

Performance Impact

• Before: O(n) database queries where n = number of todos/events with linked data
• After: Fixed 2-3 database queries total regardless of data size
• Expected improvement: 20-35% reduction in Activities panel load time

Validation

• ✅ Python syntax check passed
• ✅ Ruff linter passed with all checks
• ✅ Code review completed and feedback addressed
• ✅ CodeQL security scan completed with 0 alerts

The implementation maintains backward compatibility while significantly improving database query efficiency.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>N+1 Query in get_linked_todos() and get_linked_events()</issue_title>
<issue_description>

Metadata

• File(s): next_crm/api/activities.py:563-593, 617-622
• Category: Database/API
• Severity: High
• Effort to Fix: Medium
• Estimated Performance Gain: 20-35%

Problem Description

The get_linked_todos() function queries Event details and Event Participants inside a loop for each ToDo that has a linked event. Similarly, get_linked_events() queries Event Participants for each event. This creates N+1 query patterns that scale with the number of todos/events.

Impact Analysis

• CPU Impact: Medium - Query execution overhead
• Memory Impact: Low - Small result sets
• Database Impact: High - O(n) additional queries where n = todos with linked events
• User Experience Impact: High - Activities panel loads slowly

Code Location

# next_crm/api/activities.py:563-593 - N+1 in get_linked_todos()
for todo in todos:
    if todo.get("custom_linked_event", None):
        event = frappe.db.get_value(  # Query 1 per todo
            "Event",
            todo["custom_linked_event"],
            ["name", "sync_with_google_calendar", "google_calendar"],
        )
        if not event:
            continue
        todo["_event"] = {...}
        event_participants = frappe.db.get_all(  # Query 2 per todo
            "Event Participants",
            filters={"parent": todo["_event"]["name"]},
            fields=["reference_doctype", "reference_docname", "email"],
        )
        # ...

# next_crm/api/activities.py:617-622 - N+1 in get_linked_events()
for event in events:
    event["event_participants"] = frappe.db.get_all(  # Query per event
        "Event Participants",
        filters={"parent": event.name},
        fields=["reference_doctype", "reference_docname", "email"],
    )

Root Cause

1. Queries inside loops instead of batch fetching
2. Related data (Event, Event Participants) not fetched upfront
3. No consideration for the number of records being processed

Proposed Solution

Batch fetch Event and Event Participants data:

def get_linked_todos(name):
    meta = frappe.get_meta("ToDo")
    fields = [
        "name",
        "custom_title",
        "description",
        "allocated_to",
        "date",
        "priority",
        "status",
        "modified",
    ]
    if meta.has_field("custom_from_time"):
        fields.append("custom_from_time")
    if meta.has_field("custom_to_time"):
        fields.append("custom_to_time")
    if meta.has_field("custom_linked_event"):
        fields.append("custom_linked_event")

    todos = frappe.db.get_list(
        "ToDo",
        filters={"reference_name": name},
        fields=fields,
    )

    if not todos:
        return []

    # Collect all linked event names
    event_names = [
        todo["custom_linked_event"] 
        for todo in todos 
        if todo.get("custom_linked_event")
    ]

    # Batch fetch events
    events_map = {}
    if event_names:
        events = frappe.db.get_all(
            "Event",
            filters={"name": ["in", event_names]},
            fields=["name", "sync_with_google_calendar", "google_calendar"],
        )
        events_map = {e["name"]: e for e in events}

    # Batch fetch all event participants
    participants_map = {}
    if event_names:
        all_participants = frappe.db.get_all(
            "Event Participants",
            filters={"parent": ["in", event_names]},
            fields=["parent", "reference_doctype", "reference_docname", "email"],
        )
        for p in all_participants:
            if p["parent"] not in participants_map:
                participants_map[p["parent"]] = []
            participants_map[p["parent"]].append({
                "reference_doctype": p["reference_doctype"],
                "reference_docname": p["reference_docname"],
                "email": p["email"],
            })

    # Apply to todos
    for todo in todos:
        linked_event = todo.get("custom_linked_event")
        if linked_event and linked_event in events_map:
            event = events_map[linked_event]
            todo["_event"] = {
                "name": event["name"],
                "sync_with_google_calendar": event["sync_with_google_calendar"],
                "google_calendar": event["google_calendar"],
                "event_participants": participants_map.get(event["name"], []),
            }
        else:
            todo["_event"] = None

    return todos


def get_linked_events(name):
    events = frappe.db.get_list(
        "Event",
        filters=[["Event Participants", "reference_docname", "=", name]],
        fields=[
            "name", "subject", "description", "_assign", "starts_on", "ends_on",
            "event_category", "sync_with_google_calendar", "google_calendar",
            "status", "event_type", "modified",
        ],
...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes rtCamp/next-crm#330

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

[mrrobot47]

@copilot continue

[Copilot]

@copilot continue

I've completed the N+1 query fix implementation. The changes eliminate the query loops in both get_linked_todos() and get_linked_events() functions by batch fetching events and event participants upfront. This should provide the estimated 20-35% performance improvement.

Key changes in commit 4355547:

• Batch fetch events and participants using ["in", event_names] filters
• Use sets to avoid duplicate event names
• Create lookup maps for O(1) access
• Reduce queries from O(n) to a fixed 2-3 queries total

All linting checks and security scans passed with 0 alerts.