[JENKINS-61439] Many branches on a multibranch repo are slowed by cache lock on master

[jenkins-infra-bot]

On our project we have a large number of feature builds across several versions that run as separate builds on Jenkins. We have noticed that our multibranch builds for this have been creeping up in run times, with no immediately obvious cause in the code changes, and after some digging, it seems that even though each build is running independently, there is a singular lock for each unique remote name that each git process waits to acquire. Even though each fetch and checkout is relatively fast, this seems to lead to a huge backlog of builds in contention over the lock as the multibranch pipeline starts each build on dev before requisitioning a slave and executing the specific tasks needed.

I have been digging and not found anyone who seems to have hit the same issue so it is likely that we have some unusual setup that doesn't match with standard usage, but it also seems strange that we are locking on the remote name rather than the directory (which is more likely to have synchronisation issues).

The specific bit of the plugin that is acquiring the lock seems to be AbstractGitSCMSource.getCacheLock(). I've attached a thread dump illustrating the various threads all stacked up on the singular lock.

---

Originally reported by mellams, imported from: Many branches on a multibranch repo are slowed by cache lock on master

• status: Open
• priority: Major
• component(s): git-plugin
• resolution: Unresolved
• votes: 0
• watchers: 2
• imported: 2025-12-02

Raw content of original issue

On our project we have a large number of feature builds across several versions that run as separate builds on Jenkins. We have noticed that our multibranch builds for this have been creeping up in run times, with no immediately obvious cause in the code changes, and after some digging, it seems that even though each build is running independently, there is a singular lock for each unique remote name that each git process waits to acquire. Even though each fetch and checkout is relatively fast, this seems to lead to a huge backlog of builds in contention over the lock as the multibranch pipeline starts each build on dev before requisitioning a slave and executing the specific tasks needed.

I have been digging and not found anyone who seems to have hit the same issue so it is likely that we have some unusual setup that doesn't match with standard usage, but it also seems strange that we are locking on the remote name rather than the directory (which is more likely to have synchronisation issues).

The specific bit of the plugin that is acquiring the lock seems to be AbstractGitSCMSource.getCacheLock(). I've attached a thread dump illustrating the various threads all stacked up on the singular lock.

1 attachment

• threaddump.txt

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  I'm not sure what you mean by "remote name".  Can you explain further?
  
  I would assume that the lock is being acquired on the URL of the remote repository.  A single remote repository URL has a single cache directory on the Jenkins master.  That cache is locked for the duration of an update to retrieve the Jenkinsfile for each branch.
  
  If your repository has grown significantly or if the git cache on the Jenkins master has not recently been garbage collected, you might try performing a git gc on the specific cache directory on the master, just as an exploration to see if that would improve performance.
  
  We could consider multiple copies of the cache on the Jenkins master with some way of choosing one of the pool of caches.  That seems more complicated and has different challenges in trying to decide how deep to make the pool of caches and how to avoid disc space bloat in the pool of caches.
  
  You might also confirm that you are using the best branch source for your git provider.  If you're using GitHub, then you should use the GitHub branch source rather than the branch source provided by the git plugin.  The GitHub branch source can use REST API calls to retrieve the Jenkinsfile.  Likewise for Bitbucket, Gitea, and GitLab.  Use their higher level branch source if you're using one of those git providers.
  
  If your multibranch job definition has a "lightweight checkout" option, you might also compare the results when it is enabled and when it is disabled.  We've seen cases with large repositories where enabling lightweight checkout makes the operations slower.
  

I'm not sure what you mean by "remote name". Can you explain further?

I would assume that the lock is being acquired on the URL of the remote repository. A single remote repository URL has a single cache directory on the Jenkins master. That cache is locked for the duration of an update to retrieve the Jenkinsfile for each branch.

If your repository has grown significantly or if the git cache on the Jenkins master has not recently been garbage collected, you might try performing a git gc on the specific cache directory on the master, just as an exploration to see if that would improve performance.

We could consider multiple copies of the cache on the Jenkins master with some way of choosing one of the pool of caches. That seems more complicated and has different challenges in trying to decide how deep to make the pool of caches and how to avoid disc space bloat in the pool of caches.

You might also confirm that you are using the best branch source for your git provider. If you're using GitHub, then you should use the GitHub branch source rather than the branch source provided by the git plugin. The GitHub branch source can use REST API calls to retrieve the Jenkinsfile. Likewise for Bitbucket, Gitea, and GitLab. Use their higher level branch source if you're using one of those git providers.

If your multibranch job definition has a "lightweight checkout" option, you might also compare the results when it is enabled and when it is disabled. We've seen cases with large repositories where enabling lightweight checkout makes the operations slower.

[osherdp]

@MarkEWaite sorry for the specific tagging, but we're having pretty much the same issue and I wanted to get a maintainer here in the loop

We're having pretty much the same issue as the reporter, just that we're not using multibranch. What we're seeing is that we have a certain type of job that gets invoked a lot, and because we're using "lightweight checkout" for that job all the invocations are waiting for the same lock file. The relevant code is here:
https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/jenkins/plugins/git/GitSCMFileSystem.java#L177C60-L177C70

We can turn-off "lightweight checkout" but it will come with the cost of doing a full clone which is very costly (and will make the Jenkins controller much more busy which is a general issue for us)

Are there any other options we can investigate? Can the code be accommodated to use some other locking mechanism such that reads to the repo do not require locks and writes do?

[MarkEWaite]

Are there any other options we can investigate?

The options are suggested in my earlier comment, where I said:

you might try performing a git gc on the specific cache directory on the master, just as an exploration to see if that would improve performance.

You might also confirm that you are using the best branch source for your git provider. If you're using GitHub, then you should use the GitHub branch source rather than the branch source provided by the git plugin. The GitHub branch source can use REST API calls to retrieve the Jenkinsfile. Likewise for Bitbucket, Gitea, and GitLab. Use their higher level branch source if you're using one of those git providers.

You might also consider using a reference repository to reduce the cost of a full clone of the repository. An article includes links to the video and slides from past talks describing the technique.

You also asked:

Can the code be accommodated to use some other locking mechanism such that reads to the repo do not require locks and writes do?

There is an open pull request that proposes to reduce churn on the repository lock. you could install the incremental build of that pull request on your controller, see if it reduces the issue, and report the results in that pull request. The pull request is:

• https://github.com/jenkinsci/git-plugin/pull/3972/checks