Paginate Grants() to fix gRPC ResourceExhausted on large datasets

[c1-dev-bot]

Summary

• Add offset-based pagination to Grants() in pkg/connector/resources.go to prevent gRPC ResourceExhausted errors when a resource has a large number of grants
• Default page size of 1000 grants (~73KB per page, well under the 4MB gRPC limit); respects SDK-requested page size when provided
• Follows the established connector-side pagination pattern from CXH-1255 (baton-sql) per CE-579 engineering guidance

Details

The Grants() function previously returned all grants for a resource in a single gRPC response with no pagination. When a file contains ~77,000 direct grants, the serialized response (5.66MB) exceeds the baton-sdk's default 4MB MaxRecvMsgSize limit.

The fix uses simple offset-based pagination on the already-sorted in-memory grants slice:

• On first call (PageToken.Token is empty), starts at offset 0
• Returns up to pageSize grants per call
• Sets NextPageToken to the next offset when more grants remain
• Returns empty NextPageToken when all grants have been returned

This approach is appropriate for baton-file because all data is already loaded into memory from the file — we're only paginating the gRPC response, not the data loading.

Test plan

• Verify CI passes (build + existing tests)
• Manual test with a large Excel file containing >4MB worth of grants to confirm pagination works
• Verify small files (under page size) still return all grants in a single response

Fixes: CXH-1552

---

Automated PR Notice

This PR was automatically created by c1-dev-bot as a potential implementation.

This code requires:

• Human review of the implementation approach
• Manual testing to verify correctness
• Approval from the appropriate team before merging

[linear-code]

CXH-1552

[github-actions]

Connector PR Review: Paginate Grants() to fix gRPC ResourceExhausted on large datasets

Blocking Issues: 0 | Suggestions: 0 | Threads Resolved: 0
Review mode: full
View review run

Review Summary

This PR adds offset-based pagination to Grants() in pkg/connector/resources.go to prevent gRPC ResourceExhausted errors when a resource has a large number of grants. The implementation correctly parses the page token, clamps offset and end bounds, propagates the next page token only when more data remains, and respects the SDK-requested page size with a sensible default of 1000. Connector-side chunking of an already in-memory list is an appropriate pattern for baton-file (per C8), and the existing sort.SliceStable ensures deterministic ordering across pages.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

None.

[github-actions]

No blocking issues found.