diff --git a/docs/build.qmd b/docs/build.qmd index 44b1bee..7c67512 100644 --- a/docs/build.qmd +++ b/docs/build.qmd @@ -1 +1,189 @@ -# Build process +# Build process {#sec-build} + +Like software packages (e.g. in Rust, Python, R), the Git repository contains +the source of the package rather than the package itself. It includes the raw +input data and the code needed to process it into the final format, but before +the data package can be distributed, it must first be "built" (or "compiled") by +running the code to transform the raw data inputs to the final packaged format. + +At a high level, the directories and files involved in the build process are +(non-relevant files and directories are removed): + +``` +/ +├── .github/ +│ └── workflows/ +│ └── release.yml # Optional, depends on legal requirements +├── raw/ +├── staging/ +├── resources/ +├── releases/ +├── src/ +│ └── / +│ ├── __init__.py +│ └── build.py +├── . +├── CHANGELOG.md +├── README.md +├── LICENSE.md +└── +``` + +::: callout-note +Why do we "build" a data package? + +1. To keep a separation between the source code and raw input data contained + within the Git repository and the final data package that is released. +2. To ensure a reproducible build from input to output. +3. To keep a history of the changes made to the final data and to version the + data package for easier tracking of changes and updates. +4. To treat the data package as a formal "product" and apply rigorous and robust + software engineering and data engineering practices to its development and + release. +::: + +## Steps + +The initial steps, from the source of the data to staging, is managed by a +pipeline management tool, which should have its pipeline defined in +`src//build.py` (though some pipeline tools require a different +file name). We recommend using +[pytask](https://pytask-dev.readthedocs.io/en/stable/), which we've found works +well for developing data packages (for more details, see our [pytask for Python +workflow management](https://decisions.seedcase-project.org/why-pytask/) +decision post). All steps must be defined within this tool, from pulling the +source to raw, to processing the raw data into staging, and to processing or +extracting any metadata obtained from the source or staged data. + +The next steps are managed mainly by +[Sprout](https://sprout.seedcase-project.org/), which is our tool to handle the +processing from staging to resources and for (re-)generating the metadata file. + +The final steps for building the data package into `.tar` and `.zip` files are +handled through the [`justfile`](https://just.systems/) file, which would be +`` in the above directory structure. The `justfile`, if you use our +[template data package](https://template-data-package.seedcase-project.org/), +contains recipes for each step of the build process. + +If the data package is based on open data or on data not covered by legal +restrictions (e.g. GDPR), the build and release process can be automated through +a GitHub Actions workflow defined in `.github/workflows/release.yml`. + +```{mermaid} +%%| label: fig-build-process +%%| fig-cap: "The process for building a data package." +%%| fig-alt: A flowchart showing the process for building a data package. The +%%| process starts with the data from the source being pulled into the `raw/` +%%| directory. The data in `raw/` is then processed into `staging/`. The data +%%| in `staging/` is then joined into `resources/`. The metadata file is +%%| optionally regenerated from the code and checked against the data in +%%| `staging/`. Finally, the metadata and resources are bundled into a `.tar` +%%| and `.zip` file in `releases/`. + +flowchart TB + Source["source
(e.g., API
or database)"] -->|pull| Raw["raw/"] + Raw -->|process| Staging["staging/"] + Staging -->|join| Resources["resources/"] + Resources + Metadata["metadata
(Optionally
regenerated)"] + Metadata ---|check| Staging + Metadata & Resources -->|bundle| Build["releases/
.tar and .zip"] +``` + +::: callout-note +We plan on building a tool to automate and simplify the build steps, but for +now, you can use the structure and tools that we've set up in our [Template Data +Package](https://template-data-package.seedcase-project.org/). See @sec-setup +for details about that. +::: + +::: callout-important +It's important to note here that only the data in `raw/` and the metadata file +are saved into [Git LFS](https://git-lfs.github.com/) during the release +process. No other data artifacts or files are saved in the Git history. This is +described in more detail in @sec-release. +::: + +### Pull from source to raw + +To make sure we're getting the most up-to-date data, the first build step is to +pull from the various data sources into `raw/`. See @sec-raw and @sec-src for +details on how the raw data is organised and how the source code pulls the data +via the pipeline tool. The data is *not* processed in any way when it is pulled +from the source into `raw/`. + +### Raw to staging + +After pulling the source data into the `raw/` directory, the data (usually) +needs to be processed in some way. In particular, the data likely needs to be +reorganised so that it ends up in the correct resource location. The data in +`raw/` should be processed into `staging/` in a one-to-one mapping, so that +every `.` file in `raw/` is processed into a +`.parquet` file in `staging/`. This enables the potential for +parallel processing and to maintain a sequence between the two directories. Any +data in `staging/` must also match the contents of the metadata format. This is +all done to ensure that the data gets into the final resource correctly and +without issues. See @sec-staging, @sec-metadata, and @sec-src for more details +`staging/`. + +### Staging to resources + +At this stage, [Sprout](https://sprout.seedcase-project.org/) can take over and +process the data from `staging/` into `resources/`. All the timestamped files in +the individual `staging/` resource directories are processed and joined into a +single resource file in `resources/`. During processing, the data is checked +against the metadata. See @sec-resources, @sec-metadata, and [Sprout's +documentation](https://sprout.seedcase-project.org/) for more details on this +step. + +### Metadata (re-)generation (optional) + +For data packages using `datapackage.json` as the metadata format, Sprout will +also (re-)generate the file from the Python code. See @sec-metadata and +[Sprout's documentation](https://sprout.seedcase-project.org/) for a description +of this section. + +### Build artifacts + +At the end, the data package is built into one `_.tar` +file that contains the metadata file (e.g. `datapackage.json`), `LICENSE.md`, +`README.md`, `CHANGELOG.md`, and the resource files. If the data contains human +(especially health) data and if it is required to be on secure servers, an +additional `_.zip` file is also built with the same files +except for the data. This `.zip` file can be uploaded to public archives to +generate, for example, a [DOI](https://doi.org/) on +[Zenodo](https://zenodo.org), while the `.tar` file can remain on the server. +The `.tar` and `.zip` files are saved into a `releases/` directory ignored by +Git. + + + +## Development practices + +How does this build process impact developing the data package and the +development practices? Because of this formal build process, there are a few +things to be aware of. + +First, you shouldn't manually store any data in the Git history within the Git +LFS whenever you make changes and submit a pull request. Treat any data pulled +from sources or processed into staging or resources as temporary. Only the build +process should store any data or metadata in the Git history. + +Related to above, if the metadata format is `datapackage.json`, pull requests +should *not* contain any changes to it, nor should they contain any changes or +additions of data in `raw/`, `staging/`, or `resources/`. These files are +generated during the build process and should not be modified or added directly. +For `datapackage.json` based metadata, the metadata files within `src/` that +contain the metadata managed by [Sprout](https://sprout.seedcase-project.org/) +should be modified instead. For other metadata formats, the metadata file can be +modified directly and can be saved to the Git history. + +Commit messages should still be written in the Conventional Commits format, +though the specific commit types used are a bit different considering no data, +or metadata files if it is the `datapackage.json` format, are being modified or +saved directly. See @sec-release for more details on how Conventional Commits +are used in the release process and what commit messages to use. + +All files in `staging/`, `resources/`, and `releases/` should be ignored by Git +in the `.gitignore` file, aside from a `.gitkeep` or `README.md` file to keep +the directory structure. diff --git a/index.qmd b/index.qmd index 49bb6b8..45d88a4 100644 --- a/index.qmd +++ b/index.qmd @@ -28,9 +28,9 @@ the software package development cycle so that the final "data package" (or This guide mostly follows the [diátaxis](https://diataxis.fr/how-to-guides/) "how-to guide" style, though not strictly. It is a living and constantly evolving guide that is regularly updated as we learn and refine how we work and -develop data packages. We intend to continually update and release it with -every update to Zenodo and as GitHub releases. We don't expect this guide to -ever be considered "done". +develop data packages. We intend to continually update and release it with every +update to Zenodo and as GitHub releases. We don't expect this guide to ever be +considered "done". ::: ## Who you are @@ -67,8 +67,8 @@ the GitHub repository. ::: content-hidden -The PDF version of the guide is available in the releases page, as well as -the Zenodo archive. +The PDF version of the guide is available in the releases page, as well as the +Zenodo archive. ::: ## How the website is made