Test image

Dockerfile.testdb

@@ -0,0 +1,39 @@
+# Standalone PostgreSQL image with the DE schema migrated and synthetic,
+# anonymized test data baked in, for functional/integration testing of DE
+# services. This is separate from the root Dockerfile (the migration runner)
+# and is not part of the production build/deploy path.
+#
+# The schema and data are applied at BUILD time (see testdata/build/bake.sh) so
+# the container comes up already populated. Build:
+#   docker build -f Dockerfile.testdb -t de-database-testdb .
+# Run:
+#   docker run -d --name detest -p 5432:5432 de-database-testdb
+# Connect:
+#   postgres://de:de@localhost:5432/de?sslmode=disable
+
+FROM migrate/migrate:4 AS migrate
+
+FROM postgres:12
+
+# Bake the cluster into a path OUTSIDE the base image's declared
+# VOLUME (/var/lib/postgresql/data); writes under that volume during build are
+# discarded, which would throw away the baked data.
+ENV PGDATA=/var/lib/postgresql/dedata
+ENV POSTGRES_USER=de
+ENV POSTGRES_DB=de
+ENV POSTGRES_PASSWORD=de
+
+COPY --from=migrate /usr/local/bin/migrate /usr/local/bin/migrate
+COPY migrations /migrations
+COPY testdata/ /testdata/
+
+# initdb + migrate + load run as the postgres user; the baked data dir must be
+# owned by it for the runtime entrypoint to start cleanly.
+USER postgres
+RUN bash /testdata/build/bake.sh
+
+HEALTHCHECK --interval=10s --timeout=5s --start-period=20s --retries=5 \
+    CMD pg_isready -U de -d de || exit 1
+
+# Inherit the base postgres image's entrypoint/CMD, which starts the server.
+# On startup it finds a populated $PGDATA and serves it directly (no init).

README.md

@@ -75,6 +75,106 @@ Remove the most recent migration from the database:
 $ migrate -database "$DBURL" -path migrations down 1
 ```
 
+## Test database container image
+
+For functional and integration testing of DE services, this repository also
+builds a standalone PostgreSQL image that comes up with the DE schema already
+migrated and a small set of synthetic, anonymized test data preloaded. It is
+separate from the production migration-runner image described above (the root
+`Dockerfile`) and is **not** part of the production build/deploy path.
+
+The schema and data are baked into the image at build time, so a container starts
+up ready to use. The image, its build script, and the test data live alongside
+the migrations:
+
+- `Dockerfile.testdb` — builds the image.
+- `testdata/` — the build script and the synthetic data files (see
+  [`testdata/README.md`](testdata/README.md) for details and conventions).
+- `docker-compose.yml` — runs the database, and optionally the apps service.
+
+### Using Docker Compose
+
+```
+$ docker compose up -d de-database   # just the database (fast)
+$ docker compose up -d               # database + apps service (first build is slow)
+$ docker compose up -d --build       # force a rebuild after changes
+$ docker compose down -v             # stop and drop volumes
+```
+
+The database listens on host port `5432` and apps on `60000` by default. Override
+either if a port is already in use:
+
+```
+$ DE_DB_PORT=5440 APPS_PORT=60001 docker compose up -d
+```
+
+Other compose stacks can wait for a ready database with:
+
+```yaml
+depends_on:
+  de-database:
+    condition: service_healthy
+```
+
+#### apps service
+
+The compose stack can also run the [`apps`](https://github.com/cyverse-de/apps)
+service against the test database, so its endpoints can be exercised directly. The
+apps image builds from `../apps` by default, so that repository is expected next to
+this one (both under `.../cyverse-de/`); point elsewhere with
+`APPS_CONTEXT=/path/to/apps docker compose up -d`. Its test config is
+`testdata/apps/apps.properties`.
+
+apps boots against the test database, and with the metadata service also in the
+stack (below) its metadata-backed endpoints work too — e.g.
+`curl 'http://localhost:60000/apps/66666666-6666-6666-6666-666666666601/metadata?user=testuser01'`
+returns the seeded app AVUs. The central app listing/details endpoints, however,
+also call the **iplant-groups** and **permissions** services mid-request and error
+until those are added to the stack; reference/DB-only endpoints
+(`/apps/elements/*`, `/tool-requests`, `/reference-genomes`, `/`) work regardless.
+Most endpoints take a `user` query parameter naming a seeded user (Swagger UI at
+`/docs`). See [`testdata/apps/README.md`](testdata/apps/README.md).
+
+#### metadata service
+
+The stack also runs the [`metadata`](https://github.com/cyverse-de/metadata)
+service (built from `../metadata`, override with `METADATA_CONTEXT`) plus a
+`rabbitmq` broker it requires at startup. Because the metadata service addresses
+its tables unqualified, it connects as a dedicated `metadata` database role whose
+`search_path` resolves the `metadata` schema first (created by
+`testdata/sql/90_metadata_role.sql`). Its test config is
+`testdata/metadata/metadata.properties`; it listens on `60010` by default
+(`METADATA_PORT`). See [`testdata/metadata/README.md`](testdata/metadata/README.md).
+
+### Using Docker directly
+
+```
+$ docker build -f Dockerfile.testdb -t de-database-testdb .
+$ docker run -d --name de-database-testdb -p 5432:5432 de-database-testdb
+```
+
+### Connecting
+
+The database name, user, and password are all `de`, and the `de` account owns the
+database and all schemas:
+
+```
+postgres://de:de@localhost:5432/de?sslmode=disable
+```
+
+The container exposes a `pg_isready` health check, so you can wait for it to
+finish coming up before pointing tests at it:
+
+```
+$ until docker exec de-database-testdb pg_isready -U de -d de; do sleep 1; done
+```
+
+The preloaded data is sized to support functional/regression testing of DE
+services. For the apps service in particular, `testdata/COVERAGE.md` maps read
+endpoints to the fixtures that exercise them, and `testdata/verify/apps_smoke.sql`
+provides a sanity report plus assertions. See `testdata/README.md` for the full
+contents and conventions.
+
 [1]: https://github.com/cyverse-de/de-db
 [2]: https://github.com/cyverse-de/metadata-db
 [3]: https://github.com/cyverse-de/permissions-db

docker-compose.yml

@@ -0,0 +1,110 @@
+# Test stack for functional/integration testing of DE services. Not part of the
+# production build/deploy path.
+#
+#   docker compose up -d            # build (first run) and start
+#   docker compose up -d --build    # force a rebuild after changes
+#   docker compose down -v          # stop and drop volumes
+#
+# Services:
+#   de-database  the baked test database (schema + synthetic data)
+#   rabbitmq     message broker required by the metadata service at startup
+#   apps         the apps service, built from ../apps, pointed at de-database
+#   metadata     the metadata service, built from ../metadata
+#
+# The apps and metadata build contexts default to ../apps and ../metadata, so this
+# repo (de-database) and those repos are expected side by side under
+# .../cyverse-de/; override with APPS_CONTEXT / METADATA_CONTEXT. The apps image
+# only needs the database to boot (its service clients are lazy); metadata needs
+# both the database and rabbitmq. Many apps endpoints (the main app
+# listing/details) also call the iplant-groups and permissions services, which are
+# not yet in the stack, so those error until added. See testdata/apps/README.md
+# and testdata/metadata/README.md.
+#
+# Bring up just the database (the service builds are slow on first run):
+#   docker compose up -d de-database
+#
+# Host ports default to 5432 (db), 60000 (apps), 60010 (metadata), 5672 (rabbitmq);
+# override any if already in use:
+#   DE_DB_PORT=5440 APPS_PORT=60001 METADATA_PORT=60011 RABBITMQ_PORT=5673 docker compose up -d
+# Override the service build contexts:
+#   APPS_CONTEXT=/path/to/apps METADATA_CONTEXT=/path/to/metadata docker compose up -d
+#
+# Connect: postgres://de:de@localhost:5432/de?sslmode=disable
+#          apps at http://localhost:60000 (Swagger UI at /docs)
+#          metadata at http://localhost:60010 (Swagger UI at /docs)
+#
+# Other compose stacks can wait for a ready database with:
+#   depends_on:
+#     de-database:
+#       condition: service_healthy
+
+services:
+  de-database:
+    build:
+      context: .
+      dockerfile: Dockerfile.testdb
+    image: de-database-testdb
+    container_name: de-database-testdb
+    ports:
+      - "${DE_DB_PORT:-5432}:5432"
+    environment:
+      POSTGRES_USER: de
+      POSTGRES_DB: de
+      POSTGRES_PASSWORD: de
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "de", "-d", "de"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+
+  rabbitmq:
+    image: rabbitmq:3-management
+    container_name: de-rabbitmq-test
+    environment:
+      # RabbitMQ's built-in "guest" user only works over localhost, so define a
+      # user the metadata container can use across the compose network.
+      RABBITMQ_DEFAULT_USER: de
+      RABBITMQ_DEFAULT_PASS: de
+    ports:
+      - "${RABBITMQ_PORT:-5672}:5672"
+      - "${RABBITMQ_MGMT_PORT:-15672}:15672"
+    healthcheck:
+      test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+
+  apps:
+    build:
+      context: ${APPS_CONTEXT:-../apps}
+      dockerfile: Dockerfile
+    image: cyverse-de/apps:test
+    container_name: de-apps-test
+    depends_on:
+      de-database:
+        condition: service_healthy
+    ports:
+      - "${APPS_PORT:-60000}:60000"
+    volumes:
+      - ./testdata/apps/apps.properties:/etc/iplant/de/apps.properties:ro
+
+  metadata:
+    build:
+      context: ${METADATA_CONTEXT:-../metadata}
+      dockerfile: Dockerfile
+    image: cyverse-de/metadata:test
+    container_name: de-metadata-test
+    depends_on:
+      de-database:
+        condition: service_healthy
+      rabbitmq:
+        condition: service_healthy
+    ports:
+      - "${METADATA_PORT:-60010}:60000"
+    volumes:
+      - ./testdata/metadata/metadata.properties:/etc/iplant/de/metadata.properties:ro
+    # The metadata image's default CMD is "--help"; override it to run the server.
+    # (The apps image has no such default, so it needs no command here.)
+    command: ["--config", "/etc/iplant/de/metadata.properties"]

testdata/COVERAGE.md

@@ -0,0 +1,67 @@
+# Apps service test-data coverage
+
+This maps the `apps` service (`../apps`) read paths to the fixtures in `sql/` that
+exercise them, for building golden-master regression tests ahead of a Go rewrite.
+
+Legend: **DB** = fully exercisable with this image alone; **+meta** = also needs the
+metadata service running against this DB; **+perms** = also needs the permissions
+service; **ext** = data comes from an external system (Tapis/Agave) and is not in
+this DB at all.
+
+## Fixtures and what they add
+
+| File | Domain |
+|------|--------|
+| `00_users.sql` | 10 synthetic users |
+| `10_integration_and_tools.sql` | integration_data, container_images (incl. OSG image), tools (executable/interactive/OSG), tasks (incl. external/Agave task + pipeline + showcase tasks) |
+| `15_container_settings.sql` | container_settings, devices, volumes, ports, interactive proxy settings |
+| `20_apps.sql` | 10 apps: simple, multi-version (deleted/disabled), OSG, external, pipeline, parameters-showcase; app_versions; app_steps |
+| `22_parameters.sql` | every parameter type, parameter_values (incl. tree parent/child), file_parameters, validation_rules + arguments |
+| `25_pipelines.sql` | workflow_io_maps + input_output_mapping (2-step pipeline) |
+| `28_app_docs_refs.sql` | app_documentation, app_references |
+| `30_workspaces_categories.sql` | per-user workspace + Workspace/dev/favorites categories; Beta + Bioinformatics public categories; suggested_groups |
+| `35_tool_requests.sql` | tool_requests + tool_request_statuses |
+| `45_publication_requests.sql` | app_publication_requests + statuses; extra ratings |
+| `70_metadata.sql` | metadata AVUs incl. Beta/ontology/community AVUs |
+| `80_permissions.sql` | permissions subjects (user + group), resources, grants |
+
+## Endpoint coverage
+
+| Endpoint (read) | Tier | Fixtures |
+|-----------------|------|----------|
+| `GET /apps` (search/list, paging) | DB | 20, 22, 30, 45 (ratings), 80 (public set) |
+| `GET /apps/:sys/:id/listing` | DB | 20, 45 |
+| `GET /apps/:sys/:id` (job view) | DB | 20, 22, 15 |
+| `GET /apps/:sys/:id/details` | DB | 20, 28, 45 |
+| `GET /apps/:sys/:id/documentation` | DB | 28 |
+| `GET /apps/:sys/:id/tasks` | DB | 10, 20 |
+| `GET /apps/:sys/:id/tools` | DB | 10, 15 |
+| `GET /apps/:sys/:id/versions/...` | DB | 20 (3-version app; deleted/disabled) |
+| `GET /apps/categories` (+ counts) | DB | 30 |
+| `GET /apps/categories/:sys/:cat` | DB | 30 |
+| `GET /apps/categories/featured` (Beta) | +meta | 30, 70 (beta AVU) |
+| `is_favorite` flag in listings | DB | 30 (favorites category) |
+| `GET /apps/pipelines/:id/ui` | DB | 20, 22, 25 |
+| `GET /tools`, `GET /tools/:id` | DB | 10, 15 |
+| `GET /tools/:id/apps` | DB | 10, 20 |
+| `GET /tool-requests`, `/status-codes` | DB | 35 |
+| `GET /reference-genomes` | DB | migration-seeded `genome_reference` |
+| `GET /admin/integration-data` | DB | 10 |
+| `GET /admin/apps/publication-requests` | DB | 45 |
+| `GET /apps/elements/*` (param/tool/value types, formats, info types, rule types, data sources) | DB | migration-seeded reference data |
+| `GET /apps/:id/metadata` (AVUs) | +meta | 70 |
+| `GET /apps/communities/...`, `/apps/hierarchies/...` | +meta | 70 (community/ontology AVUs) |
+| `POST /apps/permission-lister`, `/apps/sharing` | +perms | 80 |
+| `GET /apps/hpc/...` (Tapis apps) | ext | n/a (external system) |
+
+## Notes for the rewrite
+
+- The pipeline app (`Test Pipeline 1`, `66666666-…-609`) has `step_count = 2` /
+  `task_count = 2` and an input→output mapping — use it for pipeline + multi-step
+  listing assertions.
+- The versioned app (`66666666-…-606`) has three versions; `1.0.0` is deleted and
+  `2.0.0` is disabled, so default listings should show only `3.0.0`.
+- `is_favorite` is true for `Test App 1` when listing as `testuser01` (it's in that
+  user's "Favorite Apps" category, child_index 1 of the workspace root).
+- `+meta`/`+perms` rows are seeded now but only become testable once those services
+  are added to the harness; until then, assert those endpoints separately.