Troubles with local shallow clone as an origin repository

[jimklimov]

For context, I am preparing Jenkins images for a project where pre-built pre-configured containers would act as automation servers with logging and UI for a specific business purpose, embedded on offline devices. To make the implementation similar to our other (CI) workflows, we want to use the Jenkins Shared Libraries and Pipeline job definitions from Git. However to keep the footprint small (and remove potential leaks that could have been in git histories), the prepared images use shallow clones of our CI development repositories, as of some specific commit (branch tip, tag). The run-time deployments are offline and would not see any real "git remote" systems.

So the containers with Jenkins include a location with sub-directories, where each one is a bare git repository with one or more symbolically named git references (we may deliver several branches to shuffle production/troubleshooting code variants etc.) for histories just a single commit long, and no persistently defined remotes. Initially truly shallow commits were used (with git metadata saying that we should trust that invisible histories do exist somewhere, just not here), but this proved troublesome (throwing NPE from a constructor) as explored in this ticket. A workaround is possible, however the unexplained NPE is not nice so subsequent commits will try to address it.

During this effort I started with something I use for ages for JSL development: using a local filesystem repository (my active git workspace) as the Git source for configured Global Libraries. This works cleanly with paths and file:// URI schema alike, and (may be a separate bug or not) does not stumble upon hudson.plugins.git.GitSCM.ALLOW_LOCAL_CHECKOUT property protection, at least not when configuring and checking branch names in $JENKINS_URL/configure UI (Jenkins did complain in subsequent @Library loads when instantiating jobs). It also did not seem to mind using shallow commits in the source repository for branch-based references to @Library loads.

Expanding this experience to "Pipeline from SCM" failed for me with (default) Lightweight checkouts, with job run attempts instantly logging the NPE:

Started by user admin
java.lang.NullPointerException: Cannot invoke "org.eclipse.jgit.lib.Ref.getObjectId()" because the return value of "org.eclipse.jgit.lib.Repository.findRef(String)" is null
	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem.lambda$new$0(GitSCMFileSystem.java:117)
	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem.lambda$invoke$e8567d7e$1(GitSCMFileSystem.java:184)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:28)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:83)
	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem.invoke(GitSCMFileSystem.java:184)
	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem.<init>(GitSCMFileSystem.java:117)
	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:407)
	at PluginClassLoader for scm-api//jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:219)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:123)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70)
	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:456)
Finished: FAILURE

Disabling Lightweight checkouts did not crash instantly, but stumbled on that protective property (not a show-stopper, just a point for comparison):

Started by user admin
Checking out git /var/jenkins_home/.git-seed/ci into
  /var/jenkins_home/workspace/Manage-LNexus@script/8c760ac94efe469c7e0e79342dd71dbee760c441188cadda57210bbf73f3c51e
  to read Jenkinsfile-lnexus-deliver
ERROR: Checkout of Git remote '/var/jenkins_home/.git-seed/ci' aborted because it
  references a local directory, which may be insecure. You can allow local checkouts
  anyway by setting the system property 'hudson.plugins.git.GitSCM.ALLOW_LOCAL_CHECKOUT' to true.
ERROR: Maximum checkout retry attempts reached, aborting
ERROR: Checkout of Git remote '/var/jenkins_home/.git-seed/ci' aborted because it
  references a local directory, which may be insecure. You can allow local checkouts
  anyway by setting the system property 'hudson.plugins.git.GitSCM.ALLOW_LOCAL_CHECKOUT' to true.
ERROR: Maximum checkout retry attempts reached, aborting
Finished: FAILURE

After some drilling in the filesystem with consultations from ChatGPT, the root cause seems to have been constrained to the use of shallow branch as the desired source: at least in $JENKINS_HOME/caches/git-$HASH/ directories made as part of further checkout by default (probably not needed for this setup, but whatever), the git index and sample hook files were instantiated, but the refs directory remained empty. With manual reproduction, git actively refused the shallow source (see last line):

jenkins@jenkins-b7bc9cbc-95n6j:~/caches/git-1f1cb224ce6ef770a77794222e6f2a6e$ git fetch -v origin \
   +refs/heads/staging:refs/remotes/origin/staging
remote: Enumerating objects: 148, done.
remote: Counting objects: 100% (148/148), done.
remote: Compressing objects: 100% (98/98), done.
remote: Total 148 (delta 47), reused 148 (delta 47), pack-reused 0 (from 0)
Receiving objects: 100% (148/148), 603.41 KiB | 27.43 MiB/s, done.
Resolving deltas: 100% (47/47), done.
warning: rejected refs/remotes/origin/staging because shallow roots are not allowed to be updated

$ git fsck --no-reflogs --unreachable
Checking object directories: 100% (256/256), done.
Checking objects: 100% (148/148), done.
notice: HEAD points to an unborn branch (master)
notice: No default references
dangling commit 4969235855bb3867de3f31c9c794addd3b1a839d

As the result, when the plugin makes a lookup for refs/heads/staging or refs/heads/origin/staging in the cloned repository, it finds nothing and the constructor ends up with findRef(head) == null in https://github.com/jenkinsci/git-plugin/blob/git-5.10.1/src/main/java/jenkins/plugins/git/GitSCMFileSystem.java#L117:

commitId = rev == null ? invoke((Repository repository) -> repository.findRef(head).getObjectId()) : ObjectId.fromString(rev.getHash());

called from https://github.com/jenkinsci/git-plugin/blob/git-5.10.1/src/main/java/jenkins/plugins/git/GitSCMFileSystem.java#L407:

return new GitSCMFileSystem(client, remote, Constants.R_REMOTES + remoteName + "/" + headNameResult.headName, (AbstractGitSCMSource.SCMRevisionImpl) rev);

(note that default remoteName=="origin" is valid for the cloned repo, just not for the original local one).

For our local remediation, I used git commit-tree (as replicated in the PR'ed test case) to forge a complete single-commit git history in the expected branch name, as the starting and only commit of the branch.

ChatGPT suggested that we could also

make Jenkins fetch with --no-shallow but the Git plugin does not expose this cleanly for this path. You would need to patch/configure the Git client behavior, and it is fighting the intended model.

Maybe this suggestion should be explored too. For now I plan to implement a meaningful message instead of unhelpful NPE via something like:

Ref ref = repository.findRef(head);
if (ref == null) {
    throw new IOException("Expected ref " + head + " was not created by fetch");
}

Testing done

Problem detected during setup.

Minimized and reproduced with help of AI.

Experience used to make a reproducer in the test cases added and posted here; they should hopefully confirm fixes later on.

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed