diff --git a/.config/cog.toml b/.config/cog.toml index ec6313e..ff5a2f1 100644 --- a/.config/cog.toml +++ b/.config/cog.toml @@ -1,6 +1,7 @@ from_latest_tag = true disable_changelog = true disable_bump_commit = true +ignore_merge_commits = true branch_whitelist = ["main"] pre_bump_hooks = [ # Quiet the log output of git-cliff, it is noisy. diff --git a/docs/layout.qmd b/docs/layout.qmd index 66ce05b..e05f1da 100644 --- a/docs/layout.qmd +++ b/docs/layout.qmd @@ -1 +1,502 @@ # Package layout {#sec-layout} + +Like R, Python, Rust, or any other package, a data package must follow a +specific directory and file layout and naming convention. If it doesn't follow +these, it isn't a package. In this chapter, we'll provide an overview of the +data package structure, its files and directories, and which chapters in this +book cover each of these in detail. Each section in this chapter will cover a +specific directory in the package. For details about the different tools we +mention in this chapter, see @sec-tools. + +While the specific programming language doesn't *strictly* matter, we [recommend +using +Python](https://decisions.seedcase-project.org/why-python-for-data-engineering/) +for developing data packages. For this book, we'll use Python in the guide, +which means that some files and directories will be specific to Python projects. + +::: callout-note +As we learn about and refine how to build data packages, we may include sections +on using R or Rust for building data packages. +::: + +## Root directory + +The root directory, which is the top-level directory of the package, should be +named after the package itself (``). This directory requires +certain files and directories to be present. There are only two optional +directories: `.github/` and `requests/`. See @sec-setup for more information +about creating this project structure and what recommended files should be in +the `<...>` placeholders in the tree below. + +```text +/ +├── .github/ # Optional +├── .config/ +├── src/ +├── raw/ +├── staging/ +├── resources/ +├── releases/ +├── requests/ # Optional +├── docs/ +├── . +├── pyproject.toml +├── README.md +├── LICENSE.md +├── CHANGELOG.md +├── +└── +``` + +We'll go over each directory below in individual sections, so we'll only +describe the `<...>` files in the next subsections. + +::: callout-important +A data package can have many more and different files and directories than those +listed above, however these are only the required and/or common files and +directories to effectively develop a data package. +::: + +::: content-hidden +The tree above was created using the below with this website +: + +```text +/ + .github/ + .config/ + src/ + raw/ + staging/ + resources/ + releases/ + requests/ + docs/ + . + pyproject.toml + README.md + LICENSE.md + CHANGELOG.md + + +``` +::: + +### `.` + +This is the metadata file for the package, which contains metadata about the +package itself as well as the data contained within it. The format and extension +will depend on the specific metadata standard being used. For example, a +`datapackage.json` file for the [Data Package](https://datapackage.org). While +`datapackage.json` shares the name with "data package", they are not the same +thing. The Data Package Standard (or specification) is a structured format to +storing metadata about data, but is not involved in how the data package itself +is built and developed. This file is covered in more detail in @sec-metadata. + +### `pyproject.toml` + +This contains the Python project configuration for the package, such as package +dependencies, name, authors, version, and other metadata. This file is required +for Python packages, which a data package is structured as to take advantage of +Python packaging tools and workflows. In @sec-src and @sec-config we'll describe +this file more and how to use it. + +### `README.md`, `LICENSE.md`, and `CHANGELOG.md` + +These are common files that are found in many open source projects, especially +in software projects. + +- The `README.md` file, as it name suggests, is the first file any person should + read when checking out the project. + +- The `LICENSE.md` file contains the license for the project, which is required + in order for other people to use the project. In @sec-license, we describe how + to choose a license for your project, as data packages require different + licensing compared to code or text. + +- The `CHANGELOG.md` file contains a list of changes made to the project within + each version update, which can be useful for users of the data package to + learn what things changed in specific versions. This file is covered in more + detail in @sec-build and @sec-release, as it is part of the release process + for the data package. + +### `` + +These are configuration files that may be required for the package, such as +`.gitignore`, `.editorconfig`, or `.zenodo.json`/`CITATION.cff` that are used +mostly by external tools (for those that can't store the config files in +`.config/`). We go over different types of config files that we recommend when +developing and releasing a data package in @sec-config and @sec-devex. + +### `` + +These are files related to building various aspects of the data package. The +main build file is `justfile`, which is used to manage all the different build +steps. Other build-type files include `_quarto.yml` for building a website for +the data package. We go over some of these files in @sec-build related to the +build process and in @sec-docs for building the website. + +## `src/` Python code + +Aside from the `pyproject.toml` file, the `src/` directory is where all the +Python and other build-related source files are kept. The structure of this +directory resembles a Python package, with several `__init__.py` files to inform +Python that they should be treated as part of the Python project. While we go +over the `src/` directory in more detail in @sec-src, the general structure of +this directory is shown below. + +```text +src/ +└── / + ├── / + │ ├── __init__.py + │ ├── core.py + │ └── / + │ ├── __init__.py + │ ├── .py + │ └── core.py + ├── / + │ ├── __init__.py + │ ├── core.py + │ └── / + │ ├── __init__.py + │ ├── .py + │ └── core.py + ├── common/ + │ └── __init__.py + ├── __init__.py + └── build.py +``` + +- The `` is the name of the Python package that is used to build + the data package, which is usually the same as the name of the data package + itself (using `_` instead of `-`). +- The `` and `` directories contain the Python code for + processing the metadata and data, respectively. +- The `` and `` directories are used to organize the + Python code by the source of the data and the eventual resource name. +- The `common/` directory contains Python code that is used across multiple + sources and resources. +- The `build.py` file contains the build tasks that are used to build the data + package. +- The `core.py` files within the subdirectories contain functions that are used + across specific sources and/or resources. + +::: content-hidden +The tree above was created using the below with this website +: + +```text +src/ + / + / + __init__.py + core.py + / + __init__.py + .py + core.py + / + __init__.py + core.py + / + __init__.py + .py + core.py + common/ + __init__.py + __init__.py + build.py +``` +::: + +## `raw/` data + +The `raw/` folder contains the original (unprocessed) data files from their +original source locations (e.g. database or +[ftp](https://en.wikipedia.org/wiki/File_Transfer_Protocol) server) that will be +processed and built into the data package. We strongly recommend that the data +in `raw/` is not modified in any way, aside from renaming the files and +compressing them for storage purposes. Keeping them unprocessed and unmodified +ensures a clear and reproducible history of how the data was built into the data +package. We go over the `raw/` directory in @sec-raw and @sec-src. The general +structure of the `raw/` directory is shown below. + +```text +raw/ +└── / + ├── . + └── / + └── . # For, e.g., image files +``` + +Raw data is organised into directories by the source of the data +(``), for example, from [REDCap](https://www.project-redcap.org/) +as `redcap` or [Garmin wearables](https://www.garmin.com/en-US/) as `garmin`. +Within each source, the data is saved to files or directories with the timestamp +of when the data was downloaded from the source as its name. The timestamp is in +the ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`), which is a standard format for +representing date and time. The details of whether to save to a file or +directory are covered in @sec-raw. + +::: content-hidden +The tree above was created using the below with this website +: + +```text +raw/ + / + . + / + . # For, e.g., image files +``` +::: + +## `staging/` data + +Data in `staging/` is all the data in `raw/` that has been processed and +cleaned. There should be a one-to-one mapping between the timestamped files in +`raw/` and the timestamped files in `staging/`. This ensures that the data in +`staging/` is reproducible and traceable back to the original data in `raw/`, +and to allow parallel processing of the files from `raw/` to `staging/`. Staging +is necessary as it gets the data ready to be in the correct format to be +included as a data resource and can allow metadata to be extracted, if needed, +from prepared data rather than from raw data. We go over the `staging/` +directory in @sec-staging as well as partially in @sec-build. The general +structure of the `staging/` directory is shown below. + +```text +staging/ +├── .gitignore # Don't track staging data +└── / + └── / + ├── / + │ └── . # For, e.g., image files + └── .parquet +``` + +The three differences from `raw/` are that the data in `staging/` is: + +- Saved as a [Parquet](https://parquet.apache.org/) file. +- Organised into directories by the eventual resource name (``) + that the data will be built into. A resource is a single conceptual dataset + that has some meaning to the researchers and/or users of the data package. For + example, the `redcap` source may have a resource called `demographics` that + contains the demographic data from the REDCap database such as age, gender, + and date of birth. +- Not tracked by Git, as the `raw/` data will already be tracked (during release + only), so downstream files in `staging/` don't need to be tracked as well. + +::: content-hidden +The tree above was created using the below with this website +: + +```text +staging/ + .gitignore # Don't track staging data + / + / + / + . # For, e.g., image files + .parquet +``` +::: + +## `resources/` data + +```text +resources/ +├── .gitignore # Don't track resources data +├── .parquet +├── / +│ └── . # For, e.g., image files +└── / + └── / + └── part-.parquet +``` + +::: content-hidden +The tree above was created using the below with this website +: + +```text +resources/ + .gitignore # Don't track resources data + .parquet + / + / + . # For, e.g., image files + / + / + part-.parquet +``` +::: + +## `releases/` build artifacts + +```text +releases/ +├── .gitignore # Don't track releases +├── _.tar +└── _.zip +``` + +::: content-hidden +The tree above was created using the below with this website +: + +```text +releases/ + .gitignore # Don't track releases + _.tar + _.zip +``` +::: + +## `docs/` documentation + +This directory contains the documentation for the data package. It is not only +for showing the metadata in a human-friendly way, but also for documenting +details about the package itself, how it was made, and any contributing or +operational details. It can also contain information about who uses the data and +how it is being used (e.g. based on the content in `requests/`). We strongly +recommend that this documentation is built into a website, using a tool like +[Quarto](https://quarto.org/). If it is published as a website, some files are +required to be present in the `docs/` directory, such as `index.qmd` files. Of +the directories described in this chapter, this is the only one that has no +specific structure to follow. A basic structure for `docs/` might look like the +following (if rendering the metadata into the website): + +```text +docs/ +├── index.qmd +└── metadata/ + ├── index.qmd + └── .qmd +``` + +The `metadata/` directory would contain the converted metadata from +machine-readable to human-readable, e.g., using our tool +[Flower](https://flower.seedcase-project.org). Other than that, you are free to +organise the documentation in any way that makes sense for your data package. +We'll cover documentation in @sec-docs. + +::: content-hidden +The tree above was created using the below with this website +: + +```text +docs/ + index.qmd + metadata/ + index.qmd + .qmd +``` +::: + +## `requests/` documentation (optional) + +This directory contains any requests that data package owners receive for +obtaining subsets of the data package. These requests are created from our tool +[Propagate](https://propagate.seedcase-project.org/) and are stored in the +`requests/` directory. Each individual request also contains the data subset, +though it isn't tracked by Git. The structure of this directory is shown below. +We go over requests in @sec-requests and subsets in @sec-subsets. See +[Propagate](https://propagate.seedcase-project.org/) for more details on how to +make requests and subset data. + +```text +requests/ +└── / + ├── request.yaml + └── subset/ + ├── .gitignore # Don't track subset data + └── -.tar +``` + +The `` is the name of the research project that someone is doing, +and who wants to use a subset of the data package for that project. + +::: content-hidden +The tree above was created using the below with this website +: + +```text +requests/ + / + request.yaml + subset/ + .gitignore # Don't track subset data + -.tar +``` +::: + +## `.github/` (optional) + +This directory may not exist if you don't develop the data package on GitHub (or +a similar service). We strongly recommend that you do, as it allows for much +easier collaboration and development of the data package. It also makes it +easier to build, release, and publish the data package, even if only the +metadata is published. Other services (like Codeberg or GitLab) have similar +features to GitHub, but because GitHub is so widely used and popular, we focus +on using it in this book. We cover this directory in @sec-ci, @sec-devex, and +partially in @sec-release. The structure of this directory is shown below: + +```text +.github/ +├── workflows/ +│ ├── checks.yml +│ ├── build-website.yml +│ └── release.yml # Optional +└── +``` + +The `workflows/` directory contains the GitHub Actions workflows that are used +to build, check, and (optionally) release the data package. If the data is +subject to any legal or privacy restrictions, the `release.yml` workflow won't +be used. The `` are any other files that GitHub may use and that +are helpful to development, such as a `CODEOWNERS` file. + +::: content-hidden +The tree above was created using the below with this website +: + +```text +.github/ + workflows/ + checks.yml + build-website.yml + release.yml # Optional + +``` +::: + +## `.config/` + +This directory contains all the configuration files used for the data package. +For example, since we recommend using +[Cocogitto](https://decisions.seedcase-project.org/why-semantic-release-with-cocogitto/) +and +[git-cliff](https://decisions.seedcase-project.org/why-changelog-with-git-cliff/) +for the release process, the configuration files for those tools should be kept +in this directory. Storing them in this directory keeps the root directory +cleaner and organises the configuration files in one place. It also follows the +XDG [`.config/`](https://dot-config.github.io) specification. We go over the +`.config/` directory in several sections, such as @sec-devex and @sec-config. +The directory structure is pretty simple, as it is a flat (non-nested) directory +with only configuration files: + +```text +.config/ +└── +``` + +::: content-hidden +The tree above was created using the below with this website +: + +```text +.config/ + +``` +::: diff --git a/index.qmd b/index.qmd index 49bb6b8..45d88a4 100644 --- a/index.qmd +++ b/index.qmd @@ -28,9 +28,9 @@ the software package development cycle so that the final "data package" (or This guide mostly follows the [diátaxis](https://diataxis.fr/how-to-guides/) "how-to guide" style, though not strictly. It is a living and constantly evolving guide that is regularly updated as we learn and refine how we work and -develop data packages. We intend to continually update and release it with -every update to Zenodo and as GitHub releases. We don't expect this guide to -ever be considered "done". +develop data packages. We intend to continually update and release it with every +update to Zenodo and as GitHub releases. We don't expect this guide to ever be +considered "done". ::: ## Who you are @@ -67,8 +67,8 @@ the GitHub repository. ::: content-hidden -The PDF version of the guide is available in the releases page, as well as -the Zenodo archive. +The PDF version of the guide is available in the releases page, as well as the +Zenodo archive. ::: ## How the website is made