[JENKINS-65277] git plugin left lock in caches repo, blocked all jobs using that cache

[jenkins-infra-bot]

The git plugin left a config.lock file in one of the caches folders The folder was two weeks old and was showing indications that it might have benefited from a git gc.

Because the config.lock file was left in that folder, many jobs failed (hundreds). They could not acquire a configuration lock on that repository. The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.

Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.

The directory contents looked like this when the lock was held for 7 or more hours:

-rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 FETCH_HEAD
-rw-r--r--   1 jenkins jenkins    164 Mar 22 11:24 config
-rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 config.lock
drwxr-xr-x   8 jenkins jenkins   4096 Mar 31 19:11 .
jenkins@​bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date
Wed 31 Mar 2021 07:12:05 PM MDT

The build log using command line git reported:

15:04:49   > git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git # timeout=11
15:04:49  hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git" returned status code 255:
15:04:49  stdout: 
15:04:49  stderr: error: could not lock config file .git/config: File exists
15:04:49  
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2608)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2538)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2534)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1920)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1932)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1546)
15:04:49  	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
15:04:49  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
15:04:49  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
15:04:49  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
15:04:49  	at hudson.model.ResourceController.execute(ResourceController.java:97)
15:04:49  	at hudson.model.Executor.run(Executor.java:429)
15:04:49  Finished: FAILURE

The build log using JGit reported:

5:03:01  Started by remote host Started from Groovy Console by Mark Waite with note: Groovy console started
15:03:01  Setting origin to https://github.com/MarkEWaite/jenkins-bugs.git
15:03:06  org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config. Ensure that no other process has an open file handle on the lock file /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config.lock, then you may delete the lock file and retry.
15:03:06  	at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:220)
15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:945)
15:03:06  Caused: hudson.plugins.git.GitException
15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:947)
15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
15:03:06  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
15:03:06  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
15:03:06  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
15:03:06  	at hudson.model.ResourceController.execute(ResourceController.java:97)
15:03:06  	at hudson.model.Executor.run(Executor.java:429)
15:03:06  Finished: FAILURE

---

Originally reported by markewaite, imported from: git plugin left lock in caches repo, blocked all jobs using that cache

• status: Open
• priority: Major
• component(s): git-plugin
• resolution: Unresolved
• votes: 0
• watchers: 2
• imported: 2025-12-02

Raw content of original issue

The git plugin left a config.lock file in one of the caches folders  The folder was two weeks old and was showing indications that it might have benefited from a git gc.

Because the config.lock file was left in that folder, many jobs failed (hundreds).  They could not acquire a configuration lock on that repository.  The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.

Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.

The directory contents looked like this when the lock was held for 7 or more hours:


-rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 FETCH_HEAD
-rw-r--r--   1 jenkins jenkins    164 Mar 22 11:24 config
-rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 config.lock
drwxr-xr-x   8 jenkins jenkins   4096 Mar 31 19:11 .
jenkins@bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date
Wed 31 Mar 2021 07:12:05 PM MDT


The build log using command line git reported:

15:04:49   > git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git # timeout=11
15:04:49  hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git" returned status code 255:
15:04:49  stdout: 
15:04:49  stderr: error: could not lock config file .git/config: File exists
15:04:49  
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2608)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2538)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2534)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1920)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1932)
15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1546)
15:04:49  	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
15:04:49  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
15:04:49  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
15:04:49  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
15:04:49  	at hudson.model.ResourceController.execute(ResourceController.java:97)
15:04:49  	at hudson.model.Executor.run(Executor.java:429)
15:04:49  Finished: FAILURE


The build log using JGit reported:

5:03:01  Started by remote host Started from Groovy Console by Mark Waite with note: Groovy console started
15:03:01  Setting origin to https://github.com/MarkEWaite/jenkins-bugs.git
15:03:06  org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config. Ensure that no other process has an open file handle on the lock file /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config.lock, then you may delete the lock file and retry.
15:03:06  	at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:220)
15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:945)
15:03:06  Caused: hudson.plugins.git.GitException
15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:947)
15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
15:03:06  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
15:03:06  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
15:03:06  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
15:03:06  	at hudson.model.ResourceController.execute(ResourceController.java:97)
15:03:06  	at hudson.model.Executor.run(Executor.java:429)
15:03:06  Finished: FAILURE

• environment: Jenkins 2.277.1 with git plugin 4.7.0 and other plugins as described in my docker-lfs repo dated 31 Mar 2021

[jenkins-infra-bot]

minhhungft9:

• Original comment link
• Raw content of original comment:

  Hi,
  
  I'm having the same issue. I already deleted the caches folder in our node machine but didn't help. I don't have Admin Monitor right for my Jenkins. Do you have any recommend on this?
  
  
  
  Started
  [Fri Feb 25 14:21:46 ICT 2022] Starting branch indexing...
   > git.exe --version # timeout=10
   > git --version # 'git version 2.31.1.windows.1'
  using GIT_SSH to set credentials functional
   > git.exe ls-remote --symref --  # timeout=10
   > git.exe rev-parse --resolve-git-dir  # timeout=10
  Setting origin to 
   > git.exe config remote.origin.url  # timeout=10
  ERROR: [Fri Feb 25 14:21:47 ICT 2022] Could not fetch branches from source
  [Fri Feb 25 14:21:47 ICT 2022] Finished branch indexing. Indexing took 1.3 sec
  FATAL: Failed to recompute children of aNFCNCIStack » Continuous-Integration
  hudson.plugins.git.GitException: Command "git.exe config remote.origin.url " returned status code 255:
  stdout: 
  stderr: error: could not lock config file .git/config: File exists
  
  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2639)
   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2569)
   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2565)
   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1951)
   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1963)
   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1577)
   at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
   at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
   at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:566)
   at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
   at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
   at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641)
   at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
   at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
   at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
   at hudson.model.ResourceController.execute(ResourceController.java:97)
   at hudson.model.Executor.run(Executor.java:429)
  Finished: FAILURE
  

Hi,

I'm having the same issue. I already deleted the caches folder in our node machine but didn't help. I don't have Admin Monitor right for my Jenkins. Do you have any recommend on this?

Started
[Fri Feb 25 14:21:46 ICT 2022] Starting branch indexing...
> git.exe --version # timeout=10
> git --version # 'git version 2.31.1.windows.1'
using GIT_SSH to set credentials functional
> git.exe ls-remote --symref --  # timeout=10
> git.exe rev-parse --resolve-git-dir  # timeout=10
Setting origin to 
> git.exe config remote.origin.url  # timeout=10
ERROR: [Fri Feb 25 14:21:47 ICT 2022] Could not fetch branches from source
[Fri Feb 25 14:21:47 ICT 2022] Finished branch indexing. Indexing took 1.3 sec
FATAL: Failed to recompute children of aNFCNCIStack » Continuous-Integration
hudson.plugins.git.GitException: Command "git.exe config remote.origin.url " returned status code 255:
stdout:
stderr: error: could not lock config file .git/config: File exists

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2639)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2569)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2565)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1951)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1963)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1577)
at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:566)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  minhhungft9 the locked configuration file is on the Jenkins controller, not on the Jenkins agent.  Since it appears your controller is running on Windows, you'll need to check that there are no git.exe processes that are still running and holding locks.  You may also need to stop and restart the Jenkins service for the controller to assure that open files are all closed.
  

minhhungft9 the locked configuration file is on the Jenkins controller, not on the Jenkins agent. Since it appears your controller is running on Windows, you'll need to check that there are no git.exe processes that are still running and holding locks. You may also need to stop and restart the Jenkins service for the controller to assure that open files are all closed.

[jenkins-infra-bot]

minhhungft9:

• Original comment link
• Raw content of original comment:

  Hi markewaite,
  
  Thank you for the information. After I deleted the gitconfig lock file on my Jenkins controller. The jobs can run again.
  
  
  Can you give any advices so that I won't get this kind of issues again?
  

Hi markewaite,

Thank you for the information. After I deleted the gitconfig lock file on my Jenkins controller. The jobs can run again.

Can you give any advices so that I won't get this kind of issues again?

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  minhhungft9 I have no advice to offer.  I have only seen this failure once in all the time that I've used the git plugin.  I don't know how to duplicate the problem or what caused the problem initially.
  

minhhungft9 I have no advice to offer. I have only seen this failure once in all the time that I've used the git plugin. I don't know how to duplicate the problem or what caused the problem initially.

[jenkins-infra-bot]

minhhungft9:

• Original comment link
• Raw content of original comment:

  markewaite for me, I made a mistake in my Discover Branches regular expression and there were lots of jobs running on different branches. I didn't want to stop them one by one so I deleted the whole multibranch folder on Jenkins, then this issue happened.
  
   
  

markewaite for me, I made a mistake in my Discover Branches regular expression and there were lots of jobs running on different branches. I didn't want to stop them one by one so I deleted the whole multibranch folder on Jenkins, then this issue happened.