diff --git a/.gitignore b/.gitignore index 5bd840e..9b4e6c9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ db-data/data s3-data/data +garage-data/meta +garage-data/data .env diff --git a/README.md b/README.md index a70c105..ac11f22 100644 --- a/README.md +++ b/README.md @@ -7,28 +7,117 @@ If you use or plan to use DSW, please let us know via [info@ds-wizard.org](mailt - Join our [**Discord** server](https://discord.gg/MW3H9tdMcT), where you can be notified about important updates and releases + we can discuss your issues and ideas. - Provide us feedback (what is good and bad, [feature requests](https://ideas.ds-wizard.org/), etc.) -This example is intended for **local setup and testing**. For production use there are many more things to do such as authentication, controlling exposed ports (e.g. do not expose ports of `postgres` and `minio`), data backups, or using proxy (with HTTPS and WebSocket enabled). As it is highly dependent on your use case, consult production deployment with your sysadmin or [contact us](https://ds-wizard.org/contact). +This example is intended for **local setup and testing**. For production use there are many more things to do such as authentication, controlling exposed ports (e.g. do not expose ports of `postgres` and `garage`), data backups, or using proxy (with HTTPS and WebSocket enabled). As it is highly dependent on your use case, consult production deployment with your sysadmin or [contact us](https://ds-wizard.org/contact). -## Usage +## Overview -This is an example deployment of the [Data Stewardship Wizard](https://ds-wizard.org) using [docker-compose](https://docs.docker.com/compose/). You can clone the repository, create `.env` file using `example.env` and run it with: +This is an example deployment of the [Data Stewardship Wizard](https://ds-wizard.org) using [Docker Compose](https://docs.docker.com/compose/) and [Garage](https://garagehq.deuxfleurs.fr/) as the S3-compatible object storage. -``` -$ docker-compose up -d -``` +It is intentionally set up as a **single-node local POC**: -Then visit [localhost:8080/wizard](http://localhost:8080/wizard/) and login as `albert.einstein@example.com` with password `password`. +- Garage runs in Docker on host ports `9000` (S3 API) and `9003` (Admin API) +- DSW points to `http://host.docker.internal:9000` so presigned URLs are reachable from the browser +- `create-bucket.sh` performs the one-time Garage bootstrap for this example For information on how to use Data Stewardship Wizard, visit our [guide](https://guide.ds-wizard.org). -## Important notes +## Quick Start + +1. Create the local environment file: + + ```bash + cp example.env .env + ``` + +2. Start the stack: + + ```bash + docker compose up -d + ``` + +3. Bootstrap Garage: + + ```bash + ./create-bucket.sh + ``` + +4. Open DSW: + + [http://localhost:8080/wizard](http://localhost:8080/wizard/) + +5. Log in with: + + - Email: `albert.einstein@example.com` + - Password: `password` + +## What The Bootstrap Does + +`create-bucket.sh` is intended to be safe to rerun for this local setup. It: + +- assigns the single-node Garage layout if the node still has no role +- creates the configured S3 bucket if it does not exist +- imports the configured Garage access key if it does not exist +- grants read, write, and owner permissions on the bucket to that key + +## Verification Checklist + +After setup, use this checklist to confirm the Garage-backed deployment works. + +### Infrastructure checks + +```bash +docker compose ps +docker compose logs garage --tail=100 +docker compose logs server --tail=100 +docker compose logs docworker --tail=100 +``` + +Expected results: + +- `garage` is running +- `server` creates the S3 client successfully +- `docworker` starts without S3 errors +- no authentication, signing, or region errors appear in the logs + +### Application checks + +In the DSW UI, verify: + +1. the application opens and login works +2. a project file can be uploaded +3. the uploaded file can be downloaded +4. a document preview can be generated +5. a document template asset URL works, if applicable + +If these checks pass, Garage is functioning as a drop-in S3-compatible backend for this example deployment. + +## Important Notes * Use `docker compose pull` to get newest image (hotfixes) before starting -* **Do not expose** PostgreSQL and MinIO to the internet (MinIO should be exposed only via proxy) +* **Do not expose** PostgreSQL and Garage to the internet in a public deployment (Garage should be behind your proxy, firewall, or private network setup) * When you want to use DSW publicly, **set up HTTPS proxy** (e.g. Nginx) with a certificate for your domain and change default accounts -* Set up volume mounted to PostgreSQL and MinIO containers for persistent data -* You have to create S3 bucket, either using Web UI (for MinIO, you can expose and use `http://localhost:9000`) or via client: https://docs.min.io/docs/minio-client-complete-guide.html#mb, e.g. use `create-bucket.sh` script -* Always use **strong passwords** and never use default values, **change the secrets** in `config/application.yml` (32 character string in `general.secret` and RSA private key in `general.rsaPrivateKey` via `ssh-keygen -t rsa -b 4096 -m PEM -f jwtRS256.key`) +* Set up volume mounted to PostgreSQL and Garage containers for persistent data +* Garage needs a one-time bootstrap after the stack starts. `create-bucket.sh` assigns the single-node layout, creates the bucket, imports the configured S3 key, and grants bucket permissions +* DSW uses `http://host.docker.internal:9000` as the S3 endpoint so both the DSW containers and the browser can reach the same local Garage endpoint +* Always use **strong passwords** and never use default values, **change the secrets** in `config/application.yml` and `.env` (JWT secret, RSA private key, Garage RPC/admin tokens, and S3 credentials) + +## Troubleshooting + +If something does not work: + +```bash +docker compose ps +docker compose logs garage --tail=200 +docker compose logs server --tail=200 +docker compose logs docworker --tail=200 +``` + +Common local issues: + +- Garage was started, but `create-bucket.sh` was not run yet +- `.env` values and the imported Garage key no longer match +- the server is still starting and has not reached a healthy state yet +- an old local data directory contains stale state from a previous attempt ## Security Audit diff --git a/config/application.yml b/config/application.yml index d5831b9..f9a0f30 100644 --- a/config/application.yml +++ b/config/application.yml @@ -59,12 +59,14 @@ general: database: connectionString: postgresql://postgres:postgres@postgres:5432/engine-wizard -# (!!) Change default password +# (!!) Change default credentials s3: + # Use a host-reachable endpoint so browser-facing presigned URLs work in local Docker. url: http://host.docker.internal:9000 - username: minio - password: minioPassword + username: GK0123456789abcdef01234567 + password: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef bucket: engine-wizard + region: garage # (!!) Configure SMTP mail: diff --git a/config/garage.toml b/config/garage.toml new file mode 100644 index 0000000..e4d79d9 --- /dev/null +++ b/config/garage.toml @@ -0,0 +1,17 @@ +metadata_dir = "/var/lib/garage/meta" +data_dir = "/var/lib/garage/data" +db_engine = "sqlite" + +# Single-node local POC settings. +replication_factor = 1 +rpc_bind_addr = "[::]:3901" +rpc_public_addr = "127.0.0.1:3901" + +[s3_api] +# Path-style requests work without extra DNS; root_domain only enables optional vhost-style access. +s3_region = "garage" +api_bind_addr = "[::]:3900" +root_domain = ".s3.garage.localhost" + +[admin] +api_bind_addr = "[::]:3903" diff --git a/create-bucket.sh b/create-bucket.sh index 33238a0..914d8a3 100755 --- a/create-bucket.sh +++ b/create-bucket.sh @@ -1,18 +1,55 @@ #!/bin/sh -set -a; source .env; set +a - -# (!!) Change default password -MINIO_NET="dsw-deployment-example_default" -MINIO_BUCKET="engine-wizard" -MINIO_USER="minio" -MINIO_PASS="minioPassword" - -docker run --rm --net $MINIO_NET \ - -e MINIO_BUCKET=$MINIO_BUCKET \ - -e MINIO_USER=$MINIO_USER \ - -e MINIO_PASS=$MINIO_PASS \ - --entrypoint sh minio/mc:$MINIO_VERSION -c "\ - mc alias set dswminio http://minio:9000 $MINIO_USER $MINIO_PASS && \ - mc mb dswminio/\$MINIO_BUCKET -" +set -eu + +if [ -f .env ]; then + set -a + . ./.env + set +a +fi + +GARAGE_SERVICE="${GARAGE_SERVICE:-garage}" +GARAGE_ZONE="${GARAGE_ZONE:-dc1}" +GARAGE_CAPACITY="${GARAGE_CAPACITY:-1G}" +GARAGE_KEY_NAME="${GARAGE_KEY_NAME:-dsw-engine-wizard}" +S3_URL="${S3_URL:-http://host.docker.internal:9000}" +S3_BUCKET="${S3_BUCKET:-engine-wizard}" +S3_USERNAME="${S3_USERNAME:-GK0123456789abcdef01234567}" +S3_PASSWORD="${S3_PASSWORD:-0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef}" + +garage_exec() { + docker compose exec -T "$GARAGE_SERVICE" /garage "$@" +} + +if [ -z "$(docker compose ps -q --status running "$GARAGE_SERVICE" 2>/dev/null)" ]; then + echo "Garage service '$GARAGE_SERVICE' is not running. Start the stack first with: docker compose up -d" >&2 + exit 1 +fi + +STATUS="$(garage_exec status)" +NODE_ID="$(printf '%s\n' "$STATUS" | awk '/==== HEALTHY NODES ====/{flag=1; next} flag && $1 ~ /^[0-9a-f]+$/ {print $1; exit}')" + +if [ -z "$NODE_ID" ]; then + echo "Unable to determine the Garage node ID from 'garage status'." >&2 + exit 1 +fi + +if printf '%s\n' "$STATUS" | grep -q "NO ROLE ASSIGNED"; then + garage_exec layout assign -z "$GARAGE_ZONE" -c "$GARAGE_CAPACITY" "$NODE_ID" + garage_exec layout apply --version 1 +fi + +if ! garage_exec bucket info "$S3_BUCKET" >/dev/null 2>&1; then + garage_exec bucket create "$S3_BUCKET" +fi + +if ! garage_exec key info "$S3_USERNAME" >/dev/null 2>&1; then + garage_exec key import "$S3_USERNAME" "$S3_PASSWORD" -n "$GARAGE_KEY_NAME" --yes +fi + +garage_exec bucket allow --read --write --owner "$S3_BUCKET" --key "$S3_USERNAME" + +echo "Garage bootstrap completed." +echo "S3 endpoint: $S3_URL" +echo "Bucket: $S3_BUCKET" +echo "Access key: $S3_USERNAME" diff --git a/docker-compose.yml b/docker-compose.yml index 01078d5..18bfdb5 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -4,22 +4,29 @@ services: image: datastewardshipwizard/wizard-server:${DSW_VERSION} platform: linux/amd64 restart: always + # Local demo only; put an HTTPS proxy in front for shared deployments. ports: - # (!!) Expose only for local deployment, externally use HTTPS proxy - 127.0.0.1:3000:3000 depends_on: - postgres - - minio + - garage volumes: - ./config/application.yml:/app/config/application.yml:ro extra_hosts: - host.docker.internal:host-gateway + # Keep runtime S3 values in .env so they stay aligned with Garage bootstrap. + environment: + S3_URL: ${S3_URL} + S3_USERNAME: ${S3_USERNAME} + S3_PASSWORD: ${S3_PASSWORD} + S3_BUCKET: ${S3_BUCKET} + S3_REGION: ${S3_REGION} client: image: datastewardshipwizard/wizard-client:${DSW_VERSION} restart: always + # Local demo only; put an HTTPS proxy in front for shared deployments. ports: - # (!!) Expose only for local deployment, externally use HTTPS proxy - 127.0.0.1:8080:8080 environment: API_URL: http://localhost:3000/wizard-api @@ -29,12 +36,19 @@ services: restart: always depends_on: - postgres - - minio + - garage - server volumes: - ./config/application.yml:/app/config/application.yml:ro extra_hosts: - host.docker.internal:host-gateway + # Keep runtime S3 values in .env so they stay aligned with Garage bootstrap. + environment: + S3_URL: ${S3_URL} + S3_USERNAME: ${S3_USERNAME} + S3_PASSWORD: ${S3_PASSWORD} + S3_BUCKET: ${S3_BUCKET} + S3_REGION: ${S3_REGION} mailer: image: datastewardshipwizard/mailer:${DSW_VERSION} @@ -48,39 +62,37 @@ services: postgres: image: postgres:${POSTGRES_VERSION} restart: always - # (!!) Expose only for debugging locally + # Local debug access only. ports: - 127.0.0.1:5432:5432 - # (!!) Change default password + # Change defaults before using this outside a local POC. environment: - POSTGRES_DB: engine-wizard - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - # (!!) Mount for persistent data + POSTGRES_DB: ${POSTGRES_DB} + POSTGRES_USER: ${POSTGRES_USER} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + # Uncomment one option below to persist PostgreSQL data locally. # volumes: # - db-data:/var/lib/postgresql/data # OR # - ./db-data/data:/var/lib/postgresql/data - minio: - image: minio/minio:${MINIO_VERSION} + garage: + image: dxflrs/garage:${GARAGE_VERSION} restart: always - command: server /data --console-address ":9001" - # (!!) Expose only for debugging locally, externally use HTTPS proxy (see MinIO docs) + # Publish Garage on the host so both containers and the browser can reach the same endpoint. + # Do not expose this directly in a public deployment. ports: - - 9000:9000 - - 9001:9001 - # (!!) Change default password + - 9000:3900 + - 9003:3903 + # Change these defaults before sharing or reusing this setup. environment: - MINIO_ROOT_USER: minio - MINIO_ROOT_PASSWORD: minioPassword - MINIO_VOLUMES: /data - # (!!) Mount and backup for persistent data - # volumes: - # - s3-data:/data - # OR - # - ./s3-data/data:/data + GARAGE_RPC_SECRET: ${GARAGE_RPC_SECRET} + GARAGE_ADMIN_TOKEN: ${GARAGE_ADMIN_TOKEN} + GARAGE_METRICS_TOKEN: ${GARAGE_METRICS_TOKEN} + volumes: + - ./config/garage.toml:/etc/garage.toml:ro + # Uncomment to persist Garage state between local restarts. + # - ./garage-data:/var/lib/garage # volumes: # db-data: -# s3-data: diff --git a/example.env b/example.env index 98fb379..2074f14 100644 --- a/example.env +++ b/example.env @@ -1,6 +1,25 @@ DSW_VERSION=4.29 # Versions of dependencies (see their docs) -# - possibly a migration may be needed when upgrading +# Upgrades may require migrations. POSTGRES_VERSION=17.5 -MINIO_VERSION=RELEASE.2025-05-24T17-08-30Z +GARAGE_VERSION=v2.2.0 + +# DSW DB configuration. +POSTGRES_USER=postgres +POSTGRES_PASSWORD=postgres +POSTGRES_DB=engine-wizard + +# DSW S3 configuration. +# Keep these aligned with the credentials imported by create-bucket.sh. +S3_URL=http://host.docker.internal:9000 +S3_USERNAME=GK0123456789abcdef01234567 +S3_PASSWORD=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +S3_BUCKET=engine-wizard +S3_REGION=garage + +# Garage daemon secrets. +# Change these defaults before sharing or reusing this setup. +GARAGE_RPC_SECRET=1111111111111111111111111111111111111111111111111111111111111111 +GARAGE_ADMIN_TOKEN=changeMeGarageAdminToken +GARAGE_METRICS_TOKEN=changeMeGarageMetricsToken diff --git a/garage-data/.gitkeep b/garage-data/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/garage-data/.gitkeep @@ -0,0 +1 @@ +