Expand ability to support adaptors for other repository platforms

[hlapp]

This was one of the pieces from feedback on the BOSC submission. Right now we're only pulling content from HF (models, datasets) and GitHub (code). But there are known reservations about GitHub (and HF), and there other possibilities, for example (in likely descending order of priority):

• Gitlab
• Codeberg
• Dryad (I don't think there's an "organization" equivalent?)
• Zenodo (could use a dedicated collection as the "organization" equivalent?)
• Preprint servers? (this would introduce another product type "paper"?)

It seems that previously existing model zoos have mostly transitioned to HF?

[egrace479]

Some thoughts on potential implementations:

We could certainly take a look at the Gitlab API and Codeberg's API for some docs on integration. A Google search for comparisons turned up this related question, leading to a repo that combines the three as a locally hosted tracker of projects of interest. Generalizing the code option for the Catalog may not be too complicated (it's currently hard-coded for GitHub in the code section).

The Hugging Face section has the default API URL from the config file, but it is HF-specific and then cycles through repo types (models, datasets, and spaces). I could see a config option to call Dryad or Zenodo instead.

This could be implemented through extracting each of these as a repo source module that gets called based on the config settings. If we extracted the current code and HF blocks, that could potentially provide a model for creating other options. GH and HF each have their own catalog cards as well. I don't think Dryad or Zenodo have a "like" equivalent, but the current code doesn't require that (it would just reduce functionality of those sorting options).

Papers could potentially be implemented through the HF Trending Papers (replaced papers with code) since they can be added to a collection and--more relevantly--are tracked through linkage with an org, though this does still stick with the HF model. Another option would be Google Scholar (though I think the Imageomics Google Scholar is manually curated). This feels a bit out of scope for the Catalog in that it wouldn't fit existing categories (e.g., arXiv doesn't have keywords, just subjects, nor does it have "likes"). I feel like there are probably other projects out there doing something along these lines that it would warrant more

[hlapp]

Yes I agree we should initially focus on code version control (Git) repository hosting platforms only, and defer data archives to the future when there's a more concrete use-case that can inform why and how to do that.

The use-case for integrating platforms like Gitlab and Codeberg is simple and clear: Some people and thus an organization of which they are members feel(s) strongly about GitHub's proprietary nature and/or MS corporate ownership, and have thus resolved not to use GitHub and to use alternatives instead (Gitlab being more on the hybrid side, whereas Codeberg platform software is fully open-source, and the entity is non-profit). So long as the alternatives have sufficiently similar or essentially identical features to the ones the catalog utilizes from GitHub, such organizations should not be precluded from using the Catalog.

[egrace479]

I think a first step would be to replace the GitHub API URLs (at Org fetch and additional repo fetch) with a GIT_API_BASE_URL or similar (in configs).

It looks like Codeberg should be able to be set up as a drop-in replacement (looking at the org and repo calls. It's also easy to browse repos and orgs to find examples to test.

It looks like GitLab would present a bit more complication since they have a different structure: projects are treated like GitHub repositories, and a group would be more akin to a GitHub Org. it looks like returns should still match, though I see notes about some deprecation on project returns from groups in the v5 API, so it would need this list projects limited to 20 per page.

Sidenote on browsing GitLab

I found a way to [browse](https://gitlab.com/explore/projects/active) (by following the https://gitlab.com/gitlab-com link, then clicking "Explore" in the header), but it's really [not straightforward](https://forum.gitlab.com/t/allow-general-browsing-without-signup/25245/3) (I'm getting a lot of errors from the "fixed" links on that forum).

TL;DR: I'd propose starting with the generalization for code API that has both an ORG URL and a REPO URL in the config file with a local test that it does work for Codeberg (pick an org (this one has >100 repos) and a couple additional repos). I think GitLab would require some more changes in the code structure, so that may be a secondary update (where we'd also have notes linking Org to Group and Repo to Project) that would maybe include just a line of code or a codeblock to switch to allow for GitLab use?

[egrace479]

figshare has also been used by ABC and they do have an API. It seems like there may be "institutions", certainly there are collections and projects, which may match up. This would be more akin to Zenodo, since it's DOI'd versions, not regular updates.

[hlapp]

Figshare is a digital archive (similar in notion to Dryad). Are you saying it's being (or has been) used as a data repository more along the lines of HF?

[egrace479]

It was used to archive code and a dataset for an ABC paper, so I was adding a quick note about it to this master issue.

It seems such digital archives would require an add-on module, perhaps even just a template "repo" card that could be filled in and read from a designated folder, especially since archives aren't regularly updated like GitHub or Hugging Face repos.

I think Zenodo would land somewhere in between since it would seem to also need an exclusionary pattern to avoid duplicating code repos. iHARP uses Zenodo, so spoke about this a bit with Josephine last week.

[hlapp]

I'm inclined to leave digital archives out of scope for now. Once we change that, we should start with onboarding Dryad and Zenodo (as free and/or non-profit open-source resources). Once we have a good mechanism for those, we could expand to Figshare.

[egrace479]

Agreed. This was more a note to self in case it does become in scope.