Add website preview

.github/workflows/validate.yml

@@ -21,3 +21,59 @@ jobs:
 
       - name: Validate results against results_v4 schemas
         run: python scripts/validate_results.py
+
+  # Open or update a PR in the website repo that bumps its `data/results` submodule
+  preview:
+    needs: validate
+    runs-on: ubuntu-latest
+    if: >-
+      github.event_name == 'pull_request' &&
+      github.event.pull_request.head.repo.full_name == github.repository
+    steps:
+      - name: Open/update the website preview PR
+        env:
+          GH_TOKEN: ${{ secrets.OP_BOT_TOKEN }}
+          WEBSITE: openproblems-bio/website-v3
+          PR: ${{ github.event.pull_request.number }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+          if [ -z "${GH_TOKEN:-}" ]; then
+            echo "::warning::OP_BOT_TOKEN secret is not set; skipping the website preview PR."
+            exit 0
+          fi
+          BRANCH="bot/results-pr-${PR}"
+
+          git config --global user.name  "openproblems-bot"
+          git config --global user.email "bot@openproblems.bio"
+
+          git clone --quiet "https://x-access-token:${GH_TOKEN}@github.com/${WEBSITE}" site
+          cd site
+          git checkout -B "$BRANCH" origin/main
+
+          # Point data/results at this PR's head commit.
+          git submodule sync data/results
+          git submodule update --init data/results
+          git -C data/results fetch --quiet origin "$HEAD_SHA"
+          git -C data/results checkout --quiet "$HEAD_SHA"
+          git add data/results
+
+          if git diff --cached --quiet; then
+            echo "data/results already at ${HEAD_SHA}; nothing to update."
+          else
+            git commit --quiet -m "preview: results#${PR} (data/results @ ${HEAD_SHA:0:7})"
+          fi
+          git push --force --quiet origin "$BRANCH"
+
+          # The force-push already refreshed any existing preview PR, so create one only
+          # if it's missing. "already exists" is a success, not a failure.
+          if gh pr create -R "$WEBSITE" --base main --head "$BRANCH" \
+              --title "Preview: results#${PR}" \
+              --body "Automated preview build for [openproblems-bio/results#${PR}](https://github.com/openproblems-bio/results/pull/${PR}). Bumps the \`data/results\` submodule to that PR's head. **Do not merge** — this is a throwaway preview branch." 2> create.err; then
+            echo "Opened website preview PR for $BRANCH."
+          elif grep -q "already exists" create.err; then
+            echo "Website preview PR already open for $BRANCH (refreshed by force-push)."
+          else
+            cat create.err >&2
+            exit 1
+          fi

cell_cell_communication/v1.0.0-ligand_target/task_info.json

@@ -1,6 +1,6 @@
 {
   "name": "cell_cell_communication",
-  "label": "Cell-cell communication",
+  "label": "Cell-Cell Communication",
   "commit": "c97decf07adb2e3050561d6fa9ae46132be07bef",
   "summary": "Detect interactions between ligands and target cell types",
   "description": "\nThe growing availability of single-cell data has sparked an increased\ninterest in the inference of cell-cell communication (CCC),\nwith an ever-growing number of computational tools developed for this purpose.\n\nDifferent tools propose distinct preprocessing steps with diverse\nscoring functions, that are challenging to compare and evaluate.\nFurthermore, each tool typically comes with its own set of prior knowledge.\nTo harmonize these, [Dimitrov et\nal, 2022](https://openproblems.bio/bibliography#dimitrov2022comparison) recently\ndeveloped the [LIANA](https://github.com/saezlab/liana) framework, which was used\nas a foundation for this task.\n\nThe challenges in evaluating the tools are further exacerbated by the\nlack of a gold standard to benchmark the performance of CCC methods. In an\nattempt to address this, Dimitrov et al use alternative data modalities, including\nthe spatial proximity of cell types and\ndownstream cytokine activities, to generate an inferred ground truth. However,\nthese modalities are only approximations of biological reality and come\nwith their own assumptions and limitations. In time, the inclusion of more\ndatasets with known ground truth interactions will become available, from\nwhich the limitations and advantages of the different CCC methods will\nbe better understood.\n\n**This subtask evaluates the methods' ability to predict interactions,\nthe corresponding of cytokines of which, are inferred to be active in\nthe target cell types. This subtask focuses\non the prediction of interactions from steady-state, or single-context,\nsingle-cell data.**\n\n",

cell_cell_communication/v1.0.0-source_target/task_info.json

@@ -1,6 +1,6 @@
 {
   "name": "cell_cell_communication",
-  "label": "Cell-cell communication",
+  "label": "Cell-Cell Communication",
   "commit": "c97decf07adb2e3050561d6fa9ae46132be07bef",
   "summary": "Detect interactions between source and target cell types",
   "description": "\nThe growing availability of single-cell data has sparked an increased\ninterest in the inference of cell-cell communication (CCC),\nwith an ever-growing number of computational tools developed for this purpose.\n\nDifferent tools propose distinct preprocessing steps with diverse\nscoring functions, that are challenging to compare and evaluate.\nFurthermore, each tool typically comes with its own set of prior knowledge.\nTo harmonize these, [Dimitrov et\nal, 2022](https://openproblems.bio/bibliography#dimitrov2022comparison) recently\ndeveloped the [LIANA](https://github.com/saezlab/liana) framework, which was used\nas a foundation for this task.\n\nThe challenges in evaluating the tools are further exacerbated by the\nlack of a gold standard to benchmark the performance of CCC methods. In an\nattempt to address this, Dimitrov et al use alternative data modalities, including\nthe spatial proximity of cell types and\ndownstream cytokine activities, to generate an inferred ground truth. However,\nthese modalities are only approximations of biological reality and come\nwith their own assumptions and limitations. In time, the inclusion of more\ndatasets with known ground truth interactions will become available, from\nwhich the limitations and advantages of the different CCC methods will\nbe better understood.\n\n**This subtask evaluates methods in their ability to predict interactions between\nspatially-adjacent source cell types and target cell types. This subtask focuses\non the prediction of interactions from steady-state, or single-context,\nsingle-cell data.**\n\n",

dimensionality_reduction/v1.0.0/task_info.json

@@ -1,6 +1,6 @@
 {
   "name": "dimensionality_reduction",
-  "label": "Dimensionality reduction for visualisation",
+  "label": "Dimensionality Reduction for Visualisation",
   "commit": "0a0e902bd1482e35418f7816fc91e9bc31a33126",
   "summary": "Reduction of high-dimensional datasets to 2D for visualization & interpretation",
   "description": "\nDimensionality reduction is one of the key challenges in single-cell data\nrepresentation. Routine single-cell RNA sequencing (scRNA-seq) experiments measure cells\nin roughly 20,000-30,000 dimensions (i.e., features - mostly gene transcripts but also\nother functional elements encoded in mRNA such as lncRNAs). Since its inception,\nscRNA-seq experiments have been growing in terms of the number of cells measured.\nOriginally, cutting-edge SmartSeq experiments would yield a few hundred cells, at best.\nNow, it is not uncommon to see experiments that yield over [100,000\ncells](https://openproblems.bio/bibliography#tabula2018single) or even [> 1 million\ncells.](https://openproblems.bio/bibliography#cao2020human)\n\nEach *feature* in a dataset functions as a single dimension. While each of the ~30,000\ndimensions measured in each cell contribute to an underlying data structure, the overall\nstructure of the data is challenging to display in few dimensions due to data sparsity\nand the [*\"curse of\ndimensionality\"*](https://en.wikipedia.org/wiki/Curse_of_dimensionality) (distances in\nhigh dimensional data don’t distinguish data points well). Thus, we need to find a way\nto [dimensionally reduce](https://en.wikipedia.org/wiki/Dimensionality_reduction) the\ndata for visualization and interpretation.\n\n",

foundation_models/v0.1.0/task_info.json

@@ -1,6 +1,6 @@
 {
   "name": "foundation_models",
-  "label": "Foundation models",
+  "label": "Foundation Models",
   "commit": null,
   "summary": "Modelling of single-cells to perform multiple tasks using",
   "description": "Recent developments in deep-learning have led to the creation of several 'foundation models' for single-cell data. These are large models that have been trained on data from millions of cells and am to fully capture the variability in the single-cell landscape. Typically, they use a transformer architecture [@szalata2024transformers] and undergo self-supervised pre-training using masking of parts of the input data. Trained foundation models can then be applied to a variety of downstream tasks, either by directly feeding new data into the model or by fine-tuning to better fit a new dataset or to produce a specific output. The general nature of single-cell foundation models and the large amount of data they have been trained on makes them potentially powerful tools for single-cell analysis but their performance is yet to be fully established.\n\nOpen Problems builds on existing evaluations [@boiarsky2023foundationmodels; @liu2024foundationmodels] of foundation models by incorporating them into our continuous benchmarking framework.\n\nThis overview combines results from the following benchmarks for individual tasks:\n\n- [Label projection](../label_projection)\n- [Batch Integration](../batch_integration)",

label_projection/v2.0.0/task_info.json

@@ -1,6 +1,6 @@
 {
   "name": "label_projection",
-  "label": "Label projection",
+  "label": "Label Projection",
   "commit": null,
   "summary": "Automated cell type annotation from rich, labeled reference data",
   "description": "A major challenge for integrating single cell datasets is creating matching\ncell type annotations for each cell. One of the most common strategies for\nannotating cell types is referred to as\n[\"cluster-then-annotate\"](https://www.nature.com/articles/s41576-018-0088-9)\nwhereby cells are aggregated into clusters based on feature similarity and\nthen manually characterized based on differential gene expression or previously\nidentified marker genes. Recently, methods have emerged to build on this\nstrategy and annotate cells using\n[known marker genes](https://www.nature.com/articles/s41592-019-0535-3).\nHowever, these strategies pose a difficulty for integrating atlas-scale\ndatasets as the particular annotations may not match.\n\nTo ensure that the cell type labels in newly generated datasets match\nexisting reference datasets, some methods align cells to a previously\nannotated [reference dataset](https://academic.oup.com/bioinformatics/article/35/22/4688/54802990)\nand then _project_ labels from the reference to the new dataset.\n\nHere, we compare methods for annotation based on a reference dataset.\nThe datasets consist of two or more samples of single cell profiles that\nhave been manually annotated with matching labels. These datasets are then\nsplit into training and test batches, and the task of each method is to\ntrain a cell type classifer on the training set and project those labels\nonto the test set.\n",