datamanifest[py]

Keep track of the datasets used in a scientific project. You declare your data dependencies — URLs, git repositories, checksums, formats — in a datamanifest.toml file; datamanifest downloads, verifies, extracts and loads them, and caches your own computed results with the same machinery.

• A transparent, trackable manifest. Every dataset a project depends on — URLs, DOIs, checksums (content hashes used to verify each file), formats — is listed in a single datamanifest.toml file (the manifest) that you can read directly and version with git. The format is defined by a language-agnostic spec (implemented in Python and Julia) and can be edited by hand, from code, or through the CLI.
• Fetch from a wide range of sources. Direct URLs, Zenodo/figshare and PANGAEA DOIs, git repos, object stores (s3://, gs://, …), and bulk imports from pooch, intake or DVC — all checksum-verified, extracted, and adopted in place when already on disk.
• Cache your own computed data too. The same tooling backs a @cached decorator that stores your own results with PID-lock, keyed by their inputs, to speed up calculations locally. Caching is a separate, local concern — nothing is fetched — but cached results are managed through the same CLI as datasets.
• A CLI for data download, local management and synchronization across machines. Add and download datasets, inspect and repair what's on disk, move or centralize where data is stored, and push/pull datasets and cached results between machines over rsync+ssh — all without touching your analysis code. A git-ignored state file (.datamanifest/state.toml) records where each object actually landed on this machine, keeping local location tracking separate from the portable, shareable manifest.

Installation

pip install datamanifestpy

With optional loader backends (csv, parquet, nc, yaml, fsspec, or all):

pip install "datamanifestpy[all]"

See the installation page for the per-backend details.

Quickstart

datamanifest init                  # create datamanifest.toml here
datamanifest add https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_annmean_mlo.csv --name co2
datamanifest list                  # what's tracked, and where it lives
datamanifest path co2              # resolve the on-disk path (for a script)

Then load it from your code:

import datamanifest

db = datamanifest.Database("datamanifest.toml")
df = db.load_dataset("co2")          # download on first use, then load
path = db.get_dataset_path("co2")    # just the on-disk path

Commit datamanifest.toml — the recipe of what to fetch and how. The data lives in a machine-wide shared store (deduplicated across your projects) and the private .datamanifest/ directory stays git-ignored; a collaborator clones and runs datamanifest download. See the quickstart for the full walkthrough.

Documentation

Full documentation lives at https://perrette.github.io/datamanifest/:

• Installation
• Quickstart
• Using it from your code — load_dataset, @cached, the file-less Database
• Use cases — add, repair, store, sync
• CLI reference
• Storage model · Configuration
• Adding datasets · Importing from other tools
• Language bindings · Related projects

From the same author

A few other open-source tools I maintain.

Scientific writing & data

• texmark — write scientific articles in Markdown and convert them to journal-ready LaTeX/PDF.
• papers — command-line BibTeX bibliography and PDF library manager.

Speech to Text (dictate) and Text to Speech (read-aloud) tools

• scribe — speech-to-text dictation.
• bard — text-to-speech reader.

Acknowledgments

datamanifest is a Python port of awi-esc/DataManifest.jl, written by the same author (Mahé Perrette). The Python port was implemented with assistance from Anthropic's Claude.