CDV Explorer

Modern decentralized software ecosystems evolve through crowdsourced improvement proposals (IPs) that are continuously shaped and autonomously implemented by independent actors. As a result, these ecosystems exhibit so-called Community-Driven Variability (CDV) ¹, a novel paradigm that extends beyond traditional variability-intensive systems. This tool allows to explore the proposal space of such ecosystems by providing interactive visualizations and insights about their evolution, authorship, classification, conformity, and inter-proposal relationships. For more contextual details, see accompanying tool paper ² presented at VARIABILITY'26.

👋 Introduction  |  🚀 Setup  |  ⚙️ CLI Reference  |  🛠️ Developer Notes  |  🧹 Cleanup

👋 Introduction

CDV Explorer is an ecosystem-agnostic pipeline for mining and analysing improvement proposals (IPs). At the moment, the explorer ships with active source integrations for the following two CDV-exhibiting ecosystems:

Ecosystem	IP Catalog	Source repository
Bitcoin	Bitcoin Improvement Proposals (BIPs)	bitcoin/bips
Bitcoin	SatoshiLabs Improvement Proposals (SLIPs)	satoshilabs/slips
Nostr	Nostr Implementation Possibilities (NIPs)	nostr-protocol/nips

The live site is available at seg-unibe.github.io/cdv-explorer, with a demo video on YouTube.

🚀 Setup

Requirements

Tool	Version	macOS using brew	Linux	Windows using winget
Python	3.12	brew install python@3.12	sudo apt install python3.12	winget install Python.Python.3.12
Node.js	22+ (npm bundled)	brew install node	sudo apt install nodejs npm	winget install OpenJS.NodeJS
Git	any	brew install git	sudo apt install git	winget install Git.Git

1 - Clone the repository

git clone https://github.com/SEG-UNIBE/cdv-explorer.git
cd cdv-explorer

2 - Create and activate a virtual environment

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

3 - Install dependencies

pip install -r requirements.lock.txt

Note

requirements.lock.txt pins the exact, tested versions of all transitive dependencies for reproducible pipeline runs. The direct dependencies are declared in requirements.txt; after changing them, regenerate the lock files with uv:

uv pip compile requirements.txt -o requirements.lock.txt --universal --python-version 3.12
uv pip compile requirements-dev.txt -o requirements-dev.lock.txt --universal --python-version 3.12

4 - Run the pipeline

The run command clones/updates the source repository, extracts and enriches proposal data, builds analysis artifacts, and produces React-ready exports -- all in one step.

Bitcoin (BIPs):

python main.py run -e bitcoin -s 2026-03-16 --skipllm

Nostr (NIPs):

python main.py run -e nostr -s 2026-03-16 --skipllm

Note

Omit --skipllm to also run the OpenAI-based inter-proposal relation extraction. Set the model in the ecosystem YAML under llm.model, and provide the API key via the OPENAI_API_KEY environment variable or an apikey.secret file in the project root. The pipeline picks up the file automatically when the environment variable is not set.

Snapshot date: -s is required. The pipeline resolves to the last commit whose committer timestamp falls on or before YYYY-MM-DD 23:59:59 and checks out the repository at that point.

5 - Start the React app

cd react
npm install
npm start        # Vite dev server, typically at http://localhost:5173

The frontend now uses Vite for local development and production builds. npm start and npm run dev both regenerate the snapshot index, proposal link index, and ecosystem metadata, and sync the Stage IV frontend payloads from ip_data/**/04_postprocess/ into react/public/ip_data/ before launching the dev server. Only these postprocess payloads are published — harvest, preprocess, and analysis artifacts stay local.

For a production build:

npm run build

For the frontend test suite:

npm test -- --run

⚙️ CLI Reference

CDV Explorer is driven by a Typer CLI. Run python main.py --help for a full overview.

run - execute the full pipeline

python main.py run [OPTIONS]

Options:
  -e, --ecosystem TEXT   Ecosystem slug (default: first registered)
      --source TEXT      Source slug (default: all sources for that ecosystem)
  -s, --snapshot TEXT    Snapshot date YYYY-MM-DD  [required]
      --skipllm          Skip LLM-based extraction
      --focus TEXT       Process only specific proposals (e.g. '1-9,30-44,85,A0')

Snapshot rebuild workflow:

When updating LLM extraction, dependency analysis, or other enrichment logic, rebuild only the affected proposals to avoid re-processing large snapshots:

# Rebuild analysis and postprocess for a specific proposal
python main.py run -e bitcoin -s 2026-03-16 --focus 340 --skipllm

# Rebuild analysis for multiple proposals
python main.py run -e bitcoin -s 2026-03-16 --focus 1-10,320-340 --skipllm

# Rebuild an entire snapshot (regenerate all four pipeline stages)
python main.py run -e bitcoin -s 2026-03-16 --skipllm

Note

The --focus flag skips harvest and preprocess stages, re-running only analysis and postprocess on the targeted proposals. This preserves existing enrichment (compliance checks, Git history, word clouds) while updating derived metrics.

snapshots - list available snapshots

python main.py snapshots
python main.py snapshots -e bitcoin

doctor - check the local environment

python main.py doctor

Runs read-only checks for required tooling, installed Python packages, configured ecosystem sources, snapshot artifacts, generated frontend indexes, and optional LLM credentials.

artifacts rebuild - regenerate derived artifacts from existing preprocess JSON

python main.py artifacts rebuild -e bitcoin -s 2026-03-16
python main.py artifacts rebuild -e bitcoin --all

Use this when the raw/preprocessed proposal JSON is already available and you only want to rebuild analysis and React-facing exports after changing downstream logic.

ground-truth sample-ips - prefill reviewed IPs for benchmarking

python main.py ground-truth sample-ips --wizard

This interactive helper draws a stratified sample of not-yet-reviewed IPs and writes them to ground_truth/ips_append.xlsx, from where they can be copied into the editable ground_truth/ground_truth.xlsx workbook for review. The reviewed IPs and their curated interrelations are exported to ips.csv and interrelations.csv as pipeline-friendly artifacts.

ecosystems - manage ecosystem configs

python main.py ecosystems list                # show all registered ecosystems
python main.py ecosystems show bitcoin        # dump full YAML config as JSON
python main.py ecosystems add                 # scaffold a new ecosystem YAML (interactive)
python main.py ecosystems add-source bitcoin  # add a second IP catalog to an ecosystem

🛠️ Developer Notes

Development dependencies

For local test/development work, install the dev requirements instead of the runtime-only set:

pip install -r requirements-dev.lock.txt
ruff check .
mypy
python -m pytest

CI installs from the same lock file, so local and CI environments stay identical. mypy currently runs in strict mode on a small typed slice of analysis/. Extend mypy.ini as more modules are made type-clean.

For the React frontend:

cd react
npm install
npm test -- --run
npm run build

The frontend uses Vite for bundling/dev serving and Vitest for unit tests. Build output is written to react/build to stay compatible with the existing GitHub Pages and Cloudflare deployment workflows.

Pipeline architecture

The pipeline transforms raw IP corpora into versioned, frontend-ready datasets in four stages: Harvest → Preprocess → Analysis → Postprocess. Ecosystem-specific logic is confined to the first two stages, keeping the analysis and frontend layers fully reusable across ecosystems.

Project structure

.
├── ecosystems/              # ecosystem configs (YAML) — one file per ecosystem
├── pipeline/
│   ├── harvest/             # Stage I  — ecosystem-specific: clone & snapshot checkout
│   └── preprocess/          # Stage II — ecosystem-specific: preamble extraction & enrichment
├── analysis/                # Stage III/IV — ecosystem-agnostic analysis modules & postprocess
│   ├── authorship/
│   ├── classification/
│   ├── conformity/
│   ├── dependencies/
│   ├── evolution/
│   └── wordcloud/
├── react/                   # interactive frontend (D3, PrimeReact)
└── ip_data/
    └── <ecosystem>/         # e.g. bitcoin, nostr, ...
        ├── <source>/        # e.g. bips, slips, ...
        │   ├── 01_harvest/      # raw IP documents             [gitignored]
        │   ├── 02_preprocess/   # IP object model (JSON)       ← Stage II output
        │   ├── 03_analysis/     # analysis results (CSV/JSON)  ← Stage III output
        │   └── 04_postprocess/  # frontend payloads (fetched by the web app) ← Stage IV output
        ├── _combined/           # precomputed multi-source artifacts
        └── ground_truth/        # curated benchmark (editable workbook + CSV exports)

Preprocess schema

Each proposal is stored as a JSON file under 02_preprocess/<snapshot>/. The schema has three top-level blocks: raw (verbatim preamble), meta (Git-derived history and timestamps), and insights (compliance checks, word list, status history, inter-proposal relations).

{
  "raw": {
    "preamble": [dict]
  },
  "meta": {
    "last_commit": [datetime],
    "total_commits": [int],
    "git_history": [/* ... */]
  },
  "insights": {
    "formal_compliance": [/* ... */],
    "word_list": [dict],
    "changes_in_status": [/* ... */],
    "interrelations": {
      "preamble_extracted":   [set of IPs],
      "body_extracted_regex": [set of IPs],
      "body_extracted_llm":   [set of IPs]
    }
  }
}

The exact object shapes inside interrelations are source-aware and method-specific. In particular, targets use source_slug:id keys, regex-derived entries carry occurrence counts, and LLM-assisted semantic dependency extraction runs are stored as timestamped run objects with status codes, run_id links, and per-dependency metadata. Prompt provenance is stored once per source/snapshot in the shared LLM run manifest.

Concrete examples: bip-0340.json (Schnorr Signatures) · nip-10.json (Text Notes and Threads)

Adding a new ecosystem

1. Run python main.py ecosystems add and answer the prompts — a scaffolded ecosystems/<slug>.yml is created.
2. Edit the YAML to fill in classification dimensions, conformity standards, preamble field rules, and any other config.
3. Implement the ecosystem-specific Stage I & II logic in pipeline/:
   • harvest/ — a harvester that clones or fetches the IP source and checks out a snapshot
   • preprocess/ — an extractor that parses raw documents into the canonical IP object model, and a compliance checker under preprocess/checkers/
4. Add a corresponding adapter under react/src/ecosystems/<slug>/ (copy bitcoin/ or nostr/ as a template).
5. Run python main.py run -e <slug> -s <date> --skipllm to verify the pipeline end-to-end.

Deployment

Production is deployed to GitHub Pages via .github/workflows/deploy-prod.yml after a successful CI run on main or master. Development builds are deployed to Cloudflare Pages via .github/workflows/deploy-dev.yml after a successful CI run on dev. To enable GitHub Pages on a fork, go to Settings > Pages and set the source to GitHub Actions. Both workflows build the Vite app and publish the generated react/build directory.

Releases

Releases are drafted automatically by .github/workflows/draft-release.yml whenever a v* tag is pushed:

1. On dev, bump the version in react/package.json and package-lock.json: cd react && npm version x.x.x --no-git-tag-version. Commit and push.

2. Merge dev into main (e.g. via pull request) — the CI run on main triggers the production deploy.

3. Tag the resulting merge commit on main and push the tag:

   git switch main && git pull
   git tag vx.x.x && git push origin vx.x.x

   The workflow validates that the tag matches the react/package.json version (mismatch fails the run), then creates a draft GitHub release with auto-generated notes.

4. Review the draft under Releases on GitHub, polish the notes, and publish.

Tagging the merge commit ensures the release, the deployed site, and the commit hash shown in the app header all refer to the same state.

🧹 Cleanup

Deactivate the virtual environment:

deactivate

Optionally remove individual artifacts without deleting the repository:

rm -rf .venv
rm -rf ip_data/**/01_harvest           # harvested source repos (gitignored, can be large)
rm -rf react/node_modules react/build

Or remove everything at once:

cd .. && rm -rf cdv-explorer

Footnotes

1. Bögli, R. et al. Community-driven variability: characterizing a new software variability paradigm. Autom Softw Eng 33, 67 (2026). 10.1007/s10515-026-00594-0 ↩

2. Bögli, R. and Kehrer, T. CDV-Explorer: Navigating Improvement Proposal Spectra in Decentralized OSS Ecosystems. In Companion Proc. Int'l Conf. on Software and Systems Reuse, Product Lines, and Configuration (VARIABILITY), Limassol, Cyprus, Sep. 2026. DOI: forthcoming. [Preprint] ↩