fix(github): paginateSearchWithRetry triggers 403 rate limit and disc…

[Fei-Good]

fix #33

• Data retention: Moved items and pagesFetched outside the while loop to prevent data loss during retries
• Page limit: MAX_SEARCH_PAGES = 8, with a maximum of 16 Search API requests (2 label groups), leaving 14 quota units remaining
• Rate-limiting graceful degradation: When a 403/429 error is triggered, return existing data immediately without retrying; retry only when no data is available. Additionally, the conditions for X-RateLimit-Reset have been corrected (removing the incorrect waitMs < 60_000 upper limit).

[jadeproheshan]

Moving the [items] array out of the loop and handling the 403 fallback perfectly fixes the memory clear bug during rate limits. Simple and effective.

While reviewing this, I realized we can make this even stronger. Since the GitHub Search API quota is strictly 30 requests/min, we easily hit the wall. Instead of safely failing, we can leverage the Octokit paginator to resume from where we left off (resumable pagination).

Octokit's actually accepts a page parameter. We just need to track the last successful page and pass it on the next loop, like this:

let currentPage = 1; // Track our progress

while (attempt < MAX_PAGINATION_RETRIES) {
  try {
    for await (const response of this.octokit.paginate.iterator(
      this.octokit.rest.search.issuesAndPullRequests,
      {
        q: searchQuery,
        sort: 'updated',
        order: 'desc',
        per_page: SEARCH_RESULTS_PER_PAGE,
        page: currentPage, // Resume from the dropped page
      },
    )) {
      // Handle data...
      currentPage++; // Increment heavily
      if (currentPage >= MAX_SEARCH_PAGES) break;
    }

Since these two features perfectly complement each other (your 403 handling + resuming pagination), I think we should fix them all here.

Would you mind if I push my resume logic directly to your branch so we can get it all merged at once?

[Fei-Good]

no problem

[jadeproheshan]

Because i dont have write-access to your pr,so i open a new pr #35 which combines our work.You can go ahead and review my code in that Pr and close this pr.Also, we can think through the discussion points together there.