Optimize total_count query in get_data()

[Copilot]

This PR optimizes the get_data() function in next_crm/api/doc.py by replacing inefficient len(frappe.get_list()) calls with frappe.db.count() for counting records.

Changes Made

• Line 621: Replaced len(frappe.get_list(doctype, filters=filters)) with frappe.db.count(doctype, filters=filters) for calculating total_count
• Lines 502-504: Replaced the kanban column count logic that used len(frappe.get_list()) with frappe.db.count()

Performance Impact

• Before: frappe.get_list() fetches all matching records from database, loads them into memory, then Python counts the list length
• After: frappe.db.count() executes a lightweight SQL COUNT query that only returns the count value

This eliminates:

• Unnecessary data fetching
• Memory overhead from loading all record metadata
• ORM processing overhead

Expected performance gain: 15-25% on initial page load, especially noticeable with large datasets.

Testing

• ✅ Python syntax validation passed
• ✅ Code review completed with no issues
• ✅ Security check passed with no vulnerabilities found

The changes are minimal and surgical, affecting only the count operations without altering any other functionality.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Unbounded total_count Query in get_data()</issue_title>
<issue_description>

Metadata

• File(s): next_crm/api/doc.py:623, 502-506
• Category: Database/API
• Severity: High
• Effort to Fix: Low
• Estimated Performance Gain: 15-25%

Problem Description

The get_data() function calculates total_count by running frappe.get_list() without field restrictions or limits, fetching all records just to count them. This is inefficient, especially for large datasets.

Additionally, for each kanban column, all_count is calculated the same way (line 502-506), running unbounded queries.

Impact Analysis

• CPU Impact: Medium - Frappe ORM processing overhead
• Memory Impact: High - Loads all record metadata into memory
• Database Impact: High - Full table scans or index scans without limits
• User Experience Impact: Medium - Noticeable delay on initial page load

Code Location

# next_crm/api/doc.py:623 - Unbounded query for total count
"total_count": len(frappe.get_list(doctype, filters=filters)),

# next_crm/api/doc.py:502-506 - Unbounded query per kanban column
all_count = len(
    frappe.get_list(
        doctype, filters=convert_filter_to_tuple(doctype, new_filters)
    )
)

Root Cause

Using len(frappe.get_list()) instead of frappe.db.count(). The get_list function fetches all matching records from the database and returns them as a list, then Python counts the list length. This is extremely wasteful.

Proposed Solution

Use frappe.db.count() for counting:

# Replace line 623:
# OLD:
"total_count": len(frappe.get_list(doctype, filters=filters)),

# NEW:
"total_count": frappe.db.count(doctype, filters=filters),


# Replace lines 502-506:
# OLD:
all_count = len(
    frappe.get_list(
        doctype, filters=convert_filter_to_tuple(doctype, new_filters)
    )
)

# NEW:
all_count = frappe.db.count(
    doctype, 
    filters=convert_filter_to_tuple(doctype, new_filters)
)

Implementation Steps

1. Replace len(frappe.get_list()) with frappe.db.count() at line 623
2. Replace kanban column count logic at lines 502-506
3. Verify filters are compatible with count operation
4. Test with filtered and unfiltered views
5. Consider caching counts for frequently accessed views

Related Issues

• Issue 1 in this repo (getCounts N+1 pattern)</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Unbounded total_count Query in get_data() #329

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[mrrobot47]

@copilot continue

[Copilot]

@copilot continue

I've completed the optimization. The changes are in commit 201cccb.

Changes made:

• Replaced len(frappe.get_list()) with frappe.db.count() at line 621 for total_count
• Replaced len(frappe.get_list()) with frappe.db.count() at lines 502-504 for kanban column counts

Both code review and security checks passed with no issues. The optimization eliminates unnecessary data fetching and memory overhead, improving performance especially for large datasets.