Skeleton for local supervised DVC run and pinned huggingface-regressor transformers version

[florisvdf]

Changes

This PR contains two minor changes

Fixed transformers model loading in huggingface-regressor

Model loading through AutoModelForCausalLM.from_pretrained is broken for some models with trust_remote_code=True in more recent versions of transformers. See:
huggingface/transformers#43883
huggingface/transformers#40822 (comment)

Pinning the version of transformers in huggingface-regressor to 4.46.0 fixes this.

Added skeleton for local supervised DVC run

Added a dedicated directory for local DVC runs. CONTRIBUTING.md instructs users to execute local runs with dvc repro benchmark/supervised/local/dvc.yaml --single-item. This fails since the necessary directories and dvc files are missing. This change creates a skeleton to support local execution.

The structure is as follows:

local
├── default.yaml
├── dvc.lock
├── dvc.yaml
├── metric
│   └── .gitignore
└── prediction
    └── .gitignore

This includes an updated dvc.yaml file that correctly parses relative paths.

Checklist

• I broke the PR down so that it contains a reasonable amount of changes for an effective review
• I performed a self-review of my code. Amongst other things, I have commented my code in hard-to-understand areas.
• I made corresponding changes to the documentation
• I added tests that prove my fix is effective or that my feature works
• I accounted for dependent changes to be merged and published in downstream modules

[github-actions]

✅ The benchmark has been finished successfully.
📈 The result is as below:

game,model,dataset,spearman
supervised,pls,NEIME_2019,0.0950824608163625
supervised,ritaregressor,NEIME_2019,-0.0224008614752868
zero-shot,esm,ranganathan,-0.04390739267156756

[florisvdf]

@karel-w @hredestig Care to take a look?

[github-actions]

✅ The benchmark has been finished successfully.
📈 The result is as below:

game,model,dataset,spearman
supervised,pls,NEIME_2019,0.0950824608163625
supervised,ritaregressor,NEIME_2019,0.0012309541628506054
zero-shot,esm,ranganathan,-0.04390739267156756

[florisvdf]

@hredestig @karel-w Thanks for spotting the user paths. dvc.lock is now ignored, as is .dvc/cache.

[florisvdf]

It turns out that this is not a trivial change. It seems that dvc repro (and also dvc exp run for that matter) insist on experiment tracking. Therefore, if dvc.lock is .gitignored, dvc will throw an error.

From my understanding, this behavior does not align with the intended use of local dvc runs, which we do not want to be tracked remotely (and maybe not even locally?)

So far, the best solution I have come up with is to initialize a nested dvc project under local using the --no-scm flag. This creates a new dvc project that is not tracked by git. The docs (https://doc.dvc.org/command-reference/init#initializing-dvc-without-git) describe an example use case of the --no-scm flag as:

“There is no need to keep the history at all, e.g. having a deployment automation like running a data pipeline using cron.”

One can then run dvc repro from within the nested local folder (e.g. benchmark/supervised/local, with dvc.lock, .dvc/cache and .dvc/tmp ignored in the .gitignore inside local.

However, I recognize that this is maybe not the most elegant solution, as it requires an extra nested dvc project, and only works when users run dvc repro within the nested dvc project.

Thoughts on this @hredestig @karel-w?

[hredestig]

Another approach could be to have a folder of git-submodules for each benchmark, where if you wanted to run your own benchmark, i.e. the local one, you are advised to first create that git repo somewhere and submodule it in this repo. This repo is then not concerned with tracking dvc.lock just the pipelines definitions etc. No idea if that works but would avoid the unfortunate --no-scm