WIP: Proof of concept implementation for local package caching#268
Open
tzakharko wants to merge 2 commits intor-lib:mainfrom
Open
WIP: Proof of concept implementation for local package caching#268tzakharko wants to merge 2 commits intor-lib:mainfrom
tzakharko wants to merge 2 commits intor-lib:mainfrom
Conversation
Approach: relies on built-in caching for installed packages to store file checksums for local packages and use them to determine whether the contents have changed.
Author
|
Not sure why the pipeline is failing, there are some references to invalid GitHub credentials etc... all tests pass on my local machine (macOS 12.1 with R 4.1) |
Author
|
@gaborcsardi Did you by any chance have the time to look at the patch and see whether this approach is in principle sufficient? |
Member
|
I am sorry for the long wait. I am working on pkgdepends again, and will look at this soon. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a proof of concept implementation for #261
The basic idea is to leverage the fact that the metadata of installed packages is already cached. So we don't need to do much — just inject some metadata describing the contents of the local repository (in this initial draft it's the local path + the file checksums) and do nothing if they match the already installed package. Seems to work well enough in local tests, but I cannot exclude that I missed something crucial — the package logic is fairly complex and I barely scraped the surface of the iceberg in trying to understand it.
The checksumming currently relies on
tools::md5sum()and simply concatenates the hash for all files (just to show that it works), but we probably want to usedigesthere for proper sha256 (this would mean hard dependency ondigesthowever). Otherwise we need to find a way to mix the md5 hashes somehow without introducing new package dependencies. Regarding performance: checksumming does introduce some overhead, but making a package is still much much slower, so if you local packages do not change too often, amortized savings over time can be substantial.Please tell me what you think and I'll do my best to transition it to something more production ready.