0.22.0 - rate limit being exceed

[jordanabc35]

Since our concourse picked up the latest version (0.22.0) we are seeing github v3 rate limits being exceeded. Cutting back to tag: v0.21.0 appears to be working for myself and another person from the Concourse community. Possibly the new fields being grabbed adding extra calls?

[rickardl]

@jordanbcooper Exactly, thanks for the early heads up, there's a number of changes released in v0.22.0, would it be possible for you to give some more information in the number of pipelines, etc and output from your rate limit (https://developer.github.com/v3/rate_limit/).

[jrobison-sb]

@rickardl I'm not the OP but I came here to check the issues page for the same reason as OP.

In our case we're hitting the abuse rate limit, which I gather is separate from the usual rate limits shown in https://developer.github.com/v3/rate_limit/. Specifically the error in the check step is:

run check step: run check step: check: resource script '/opt/resource/check []' failed: exit status 1

stderr:
2020/10/22 20:29:57 check failed: failed to get last commits: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://developer.github.com/v3/#abuse-rate-limits\",\n  \"message\": \"You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\"\n}\n"

We're doing 7 get steps (total, across 3 pipelines), with check_every: 2m on each of them. Most or all of them use a path fileglob like:

paths:
  - foo/bar/*.baz

I'm not sure if the /rate_limit stats are relevant since we're hitting the abuse rate limit, but since you asked here's what we have:

{
  "resources": {
    "core": {
      "limit": 5000,
      "used": 39,
      "remaining": 4961,
      "reset": 1603400675
    },
    "search": {
      "limit": 30,
      "used": 0,
      "remaining": 30,
      "reset": 1603398654
    },
    "graphql": {
      "limit": 5000,
      "used": 206,
      "remaining": 4794,
      "reset": 1603402033
    },
    "integration_manifest": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1603402194
    },
    "source_import": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1603398654
    },
    "code_scanning_upload": {
      "limit": 500,
      "used": 0,
      "remaining": 500,
      "reset": 1603402194
    }
  },
  "rate": {
    "limit": 5000,
    "used": 39,
    "remaining": 4961,
    "reset": 1603400675
  }
}

[rickardl]

I got a pending implementation for #237 to mitigate this, I'm waiting for some feedback from @itsdalmo

[jmccann]

I believe core issue is too costly of query being ran due to not filtering any PRs as part of initial query

#238 (comment)
#238 (comment)

I think this was introduced by 8aa1ece / #226

Easy fix would be to try and revert that PR?

[jhosteny]

An alternative to rolling back that PR would be to ensure that this line matches the requested states, which I believe would solve the issue if you are using the default.

I'm not sure if there's a way to limit the query to the last N PRs (or filter on time, either absolute or delta from now()), but perhaps that would allow this to work with a large number of closed / merged PRs.