[Bug][GitExtractor] Extraction fails for a non-existent PR

[michelengelen]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The gitextractor started to fail for one of our repos recently.
It seems the reason is that it tries to collect data for a PR that does not exist (anymore) and after 3 retries exits with a failure message:

time="2026-03-29 12:33:22" level=error msg=" [pipeline service] [pipeline #828] [task #16537] attached stack trace\n\t  -- stack trace:\n\t  | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks\n\t  | \t/app/core/runner/run_task.go:333\n\t  | [...repeated from below...]\n\tWraps: (2) subtask Collect Pull Requests ended unexpectedly\n\tWraps: (3) attached stack trace\n\t  -- stack trace:\n\t  | github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync\n\t  | \t/app/helpers/pluginhelper/api/worker_scheduler.go:173\n\t  | github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*ApiCollector).Execute\n\t  | \t/app/helpers/pluginhelper/api/api_collector.go:206\n\t  | github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute\n\t  | \t/app/helpers/pluginhelper/api/api_collector_stateful.go:97\n\t  | github.com/apache/incubator-devlake/plugins/github/tasks.CollectApiPullRequests\n\t  | \t/app/plugins/github/tasks/pr_collector.go:140\n\t  | github.com/apache/incubator-devlake/core/runner.runSubtask\n\t  | \t/app/core/runner/run_task.go:425\n\t  | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks\n\t  | \t/app/core/runner/run_task.go:330\n\t  | github.com/apache/incubator-devlake/core/runner.RunPluginTask\n\t  | \t/app/core/runner/run_task.go:165\n\t  | github.com/apache/incubator-devlake/core/runner.RunTask\n\t  | \t/app/core/runner/run_task.go:139\n\t  | github.com/apache/incubator-devlake/server/services.runTaskStandalone\n\t  | \t/app/server/services/task_runner.go:114\n\t  | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1\n\t  | \t/app/server/services/task.go:187\n\t  | runtime.goexit\n\t  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\n\tWraps: (4) Retry exceeded 3 times calling repos/mui/material-ui/pulls/47788. The last error was: Http DoAsync error calling [method:GET path:repos/mui/material-ui/pulls/47788 query:map[]]. Response: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/pulls/pulls#get-a-pull-request\",\"status\":\"404\"} (404)\n\tError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError"

I don't know how it even gets the id for this PR, since the list call I tried does not even return it.

gh pr list --limit 1000 --json number,title,state,author,createdAt does not include the PR in question.

gh pr view 47788 --json number,title,state,author,body,createdAt,mergedAt,files returns: GraphQL: Could not resolve to a PullRequest with the number of 47788. (repository.pullRequest)

What do you expect to happen

The extractor should finish and skip 404 status silently (maybe with a warning), but not fail the collection altogether

How to reproduce

---

Anything else

every time

Version

v1.0.3-beta10

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

This is a confirmed bug. The PR collector has a two-stage collection process: Stage 1 lists PRs via pagination, and Stage 2 re-fetches individual details for all previously-collected PRs that aren't in state = 'closed' [1]. Your deleted/transferred PR still has a record in DevLake's DB from a prior sync, so Stage 2 tries to fetch it by number via repos/{{ .Params.Name }}/pulls/{{ .Input.Number }} [2] — that's where the 404 comes from.

The issue is that the CollectUnfinishedDetails stage is missing an AfterResponse handler to gracefully skip 404s [3]. Without it, the default behavior retries 404s 3 times and then crashes the subtask [4].

The fix is a one-liner. An ignoreHTTPStatus404 helper already exists in shared.go and is used by other GitHub collectors like cicd_job_collector.go [5] and account_collector.go [6]. It just needs to be added to the CollectUnfinishedDetails block in pr_collector.go:

FinalizableApiCollectorCommonArgs: helper.FinalizableApiCollectorCommonArgs{
    UrlTemplate:   "repos/{{ .Params.Name }}/pulls/{{ .Input.Number }}",
    AfterResponse: ignoreHTTPStatus404,  // ← add this line
    ResponseParser: func(res *http.Response) ([]json.RawMessage, errors.Error) {

Workaround until this is fixed: you can manually delete the offending PR record from the github_pull_requests table in DevLake's database (where number = 47788 and the matching repo_id/connection_id), so Stage 2 no longer tries to re-fetch it.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[michelengelen]

@dosu, thanks for the advice. Could you build a short query that deletes the related entries in one go?

[dosubot]

Sure! Since the child tables reference the PR by its internal github_id (not the human-visible number), you'll first need to look up the github_id. Here's a single script you can run against DevLake's database — just replace <connection_id> and <repo_id> with your actual values:

-- Step 1: Find the PR's internal github_id
SET @gh_id = (
  SELECT github_id FROM _tool_github_pull_requests
  WHERE connection_id = <connection_id> AND repo_id = <repo_id> AND number = 47788
);

-- Step 2: Delete child records
DELETE FROM _tool_github_pull_request_comments WHERE connection_id = <connection_id> AND pull_request_id = @gh_id;
DELETE FROM _tool_github_pull_request_commits  WHERE connection_id = <connection_id> AND pull_request_id = @gh_id;
DELETE FROM _tool_github_pull_request_labels   WHERE connection_id = <connection_id> AND pull_id = @gh_id;
DELETE FROM _tool_github_pull_request_reviews  WHERE connection_id = <connection_id> AND pull_request_id = @gh_id;
DELETE FROM _tool_github_pull_request_issues   WHERE connection_id = <connection_id> AND pull_request_id = @gh_id;
DELETE FROM _tool_github_reviewers             WHERE connection_id = <connection_id> AND pull_request_id = @gh_id;

-- Step 3: Delete the PR itself
DELETE FROM _tool_github_pull_requests         WHERE connection_id = <connection_id> AND github_id = @gh_id;

Note: the labels table uses pull_id instead of pull_request_id — that's intentional in the schema [1].

You can find your connection_id and repo_id from the DevLake config UI, or by querying SELECT connection_id, repo_id, number FROM _tool_github_pull_requests WHERE number = 47788;. Back up your DB before running this, just in case!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}