feat: add CI examples for gating builds on axe Watcher results

results/README.md

@@ -0,0 +1,113 @@
+# Gating CI on axe Watcher results
+
+The framework examples in this repo (see [`js/`](../js) and [`java/`](../java))
+show how to **run** an axe Watcher scan. These examples are one **optional**
+way to work with a run's results afterward: pulling them into CI with the
+[`axe-watcher-results`](https://github.com/dequelabs/axe-watcher-results) CLI
+and turning them into a **pass/fail build gate**, so a build can fail on an
+accessibility regression.
+
+| Provider                   | Example                                                        |
+| -------------------------- | -------------------------------------------------------------- |
+| GitHub Actions             | [`github-actions/a11y-gate.yml`](github-actions/a11y-gate.yml) |
+| GitLab CI                  | [`gitlab-ci/.gitlab-ci.yml`](gitlab-ci/.gitlab-ci.yml)         |
+| Any (shell)                | [`shell/gate.sh`](shell/gate.sh)                               |
+| Any (shell), by session ID | [`shell/session-results.sh`](shell/session-results.sh)         |
+
+## The gating model
+
+The CLI retrieves the axe Watcher run for a commit and **reflects the outcome
+in its exit code** — so a CI step passes or fails based on that code:
+
+```sh
+axe-watcher-results sessions get "$AXE_PROJECT_ID" "$COMMIT_SHA" --format=json
+```
+
+Each example captures the exit code, prints a clear message per outcome, and
+archives the JSON output as a build artifact.
+
+> The provider examples gate on a **commit SHA** because that is the
+> identifier CI already provides (GitHub `github.event.pull_request.head.sha`,
+> GitLab `$CI_COMMIT_SHA`). `sessions get` also accepts a **session UUID** —
+> see [By session ID](#by-session-id).
+
+## By session ID
+
+`sessions get` accepts a **session UUID** as well as a commit SHA. This is
+useful when a step already knows the exact session to check — for example a
+scan step that emits its session ID — so you gate on that session directly
+instead of resolving one from the commit:
+
+```sh
+axe-watcher-results sessions get "$AXE_PROJECT_ID" "$SESSION_ID" --format=json
+```
+
+The exit-code gating is identical to the commit-SHA path. Two session-only
+options:
+
+- `--detail=summary` (default) — the counts report, rendered per `--format`.
+- `--detail=full` — streams the raw axe Common Export Format JSON to stdout
+  (ignores `--format`), handy to archive the complete result set.
+
+See [`shell/session-results.sh`](shell/session-results.sh) for a runnable
+version. The provider examples above use a commit SHA because CI supplies one
+automatically; swap in a session UUID wherever you have one.
+
+## Exit codes
+
+| Code | Meaning                             | How the examples treat it |
+| ---- | ----------------------------------- | ------------------------- |
+| 0    | Success (under threshold)           | Pass the step             |
+| 2    | Missing required argument / env var | Fail (misconfiguration)   |
+| 3    | Invalid argument value              | Fail (misconfiguration)   |
+| 9    | Server error from DevHub            | Fail, surfaced clearly    |
+| 10   | Accessibility threshold exceeded    | **Fail — the core gate**  |
+| 11   | Session polling timed out           | Fail, surfaced clearly    |
+| 12   | No comparison data for this commit  | Policy choice (see below) |
+
+**Exit 12 — no comparison data.** The commit has scan results but the server
+has nothing to diff them against. The examples **fail by default** and print
+how to recover:
+
+- re-run the axe Watcher scan with `CI=true` (marks the session canonical, so
+  it self-compares), or
+- seed a baseline by scanning this commit again, or any earlier commit on the
+  same branch.
+
+Each example shows, in a comment, how to **warn instead of fail** (exit 0) if
+your team prefers a softer policy while baselines are being established.
+
+## Installing the CLI
+
+Download the released binary for your runner's OS/arch from Agora. Each version
+folder holds one binary per platform —
+`axe-watcher-results/<version>/<os>/<arch>/axe-watcher-results` (`.exe` on
+Windows) — for `linux/amd64`, `linux/arm64`, `darwin/arm64`, and
+`windows/amd64`, alongside a `SHA256SUMS` manifest to verify the download and
+the generated `usage.md`:
+
+```sh
+# TODO: set AGORA_BASE_URL to the published Agora repository base once the
+# publish location is finalized (dequelabs/axe-watcher-results#36).
+AGORA_BASE_URL="https://agora.dequecloud.com/artifactory/<repository>"
+CLI_VERSION="1.0.0" # the axe-watcher-results release to install
+os="$(uname -s | tr '[:upper:]' '[:lower:]')" # linux or darwin
+arch="$(uname -m)"
+case "$arch" in x86_64) arch=amd64 ;; aarch64) arch=arm64 ;; esac
+
+curl -fsSL \
+  "$AGORA_BASE_URL/axe-watcher-results/$CLI_VERSION/$os/$arch/axe-watcher-results" \
+  -o axe-watcher-results
+chmod +x axe-watcher-results
+# Then put ./axe-watcher-results on your PATH. The CI examples do this per
+# provider — e.g. the GitHub Actions example installs into "$RUNNER_TEMP/bin"
+# and appends it to "$GITHUB_PATH".
+```
+
+## Secrets
+
+`AXE_DEVHUB_API_KEY` is your axe Developer Hub API key, available from your
+axe account settings — always source it from your provider's secret store —
+GitHub Actions `secrets`, GitLab **masked** (and protected) CI/CD variables —
+never inline it in the pipeline file. `AXE_PROJECT_ID` is the project UUID and
+can be a plain variable.

results/github-actions/a11y-gate.yml

@@ -0,0 +1,71 @@
+# axe Watcher accessibility gate for GitHub Actions.
+#
+# Prerequisites (repository settings):
+#   - Secret  AXE_DEVHUB_API_KEY  — your axe Developer Hub API key.
+#   - Variable AXE_PROJECT_ID     — the project UUID.
+name: Accessibility Gate
+on:
+  pull_request:
+
+# The gate needs no repository write access; keep permissions minimal.
+permissions:
+  contents: read
+
+jobs:
+  a11y-gate:
+    name: axe Watcher gate
+    runs-on: ubuntu-latest
+    steps:
+      # Download the released CLI binary for this runner (ubuntu-latest →
+      # linux/amd64) from Agora.
+      # TODO: set AGORA_BASE_URL to the published Agora repository base once
+      # the publish location is finalized (dequelabs/axe-watcher-results#36).
+      - name: Install axe-watcher-results
+        env:
+          AGORA_BASE_URL: https://agora.dequecloud.com/artifactory/<repository>
+          CLI_VERSION: 1.0.0 # the axe-watcher-results release to install
+        run: |
+          mkdir -p "$RUNNER_TEMP/bin"
+          curl -fsSL \
+            "$AGORA_BASE_URL/axe-watcher-results/$CLI_VERSION/linux/amd64/axe-watcher-results" \
+            -o "$RUNNER_TEMP/bin/axe-watcher-results"
+          chmod +x "$RUNNER_TEMP/bin/axe-watcher-results"
+          echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
+
+      - name: Gate on the accessibility threshold
+        env:
+          AXE_DEVHUB_API_KEY: ${{ secrets.AXE_DEVHUB_API_KEY }}
+          AXE_PROJECT_ID: ${{ vars.AXE_PROJECT_ID }}
+          # On pull_request, github.sha is the merge commit — use the PR head
+          # SHA, which is the commit axe Watcher actually scanned.
+          COMMIT_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          # Capture the exit code so each outcome gets a clear message.
+          set +e
+          axe-watcher-results sessions get "$AXE_PROJECT_ID" "$COMMIT_SHA" --format=json >result.json
+          code=$?
+          set -e
+
+          cat result.json
+
+          case "$code" in
+            0)  echo "Accessibility gate passed." ;;
+            10) echo "Accessibility threshold exceeded — failing the build."; exit 1 ;;
+            12) echo "No comparison data for commit ${COMMIT_SHA}."
+                echo "Re-run the axe Watcher scan with CI=true, or seed a baseline."
+                # Policy choice: fail. To warn instead, replace with: exit 0
+                exit 1 ;;
+            11) echo "Timed out waiting for the session to finish processing."; exit 1 ;;
+            9)  echo "axe Watcher DevHub returned a server error."; exit 1 ;;
+            2|3) echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags."; exit 1 ;;
+            *)  echo "axe-watcher-results exited with code ${code}."; exit 1 ;;
+          esac
+
+      # Archive the JSON result for the build logs, even when the gate fails.
+      - name: Upload result JSON
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: axe-watcher-result
+          path: result.json
+          if-no-files-found: warn

results/gitlab-ci/.gitlab-ci.yml

@@ -0,0 +1,43 @@
+# axe Watcher accessibility gate for GitLab CI.
+#
+# Prerequisites (Settings > CI/CD > Variables):
+#   - AXE_DEVHUB_API_KEY  — your axe Developer Hub API key. Mark it Masked
+#     (and Protected). Never inline it here.
+#   - AXE_PROJECT_ID      — the project UUID (a plain variable is fine).
+accessibility-gate:
+  # A small image with curl is all that is needed to fetch the released binary.
+  image: alpine:3
+  # TODO: set AGORA_BASE_URL to the published Agora repository base once the
+  # publish location is finalized (dequelabs/axe-watcher-results#36).
+  variables:
+    AGORA_BASE_URL: https://agora.dequecloud.com/artifactory/<repository>
+    CLI_VERSION: '1.0.0' # the axe-watcher-results release to install
+  script:
+    - apk add --no-cache curl
+    - curl -fsSL "$AGORA_BASE_URL/axe-watcher-results/$CLI_VERSION/linux/amd64/axe-watcher-results" -o /usr/local/bin/axe-watcher-results
+    - chmod +x /usr/local/bin/axe-watcher-results
+    - |
+      # Capture the exit code so each outcome gets a clear message.
+      set +e
+      axe-watcher-results sessions get "$AXE_PROJECT_ID" "$CI_COMMIT_SHA" --format=json >result.json
+      code=$?
+      set -e
+
+      cat result.json
+
+      case "$code" in
+        0)  echo "Accessibility gate passed." ;;
+        10) echo "Accessibility threshold exceeded — failing the build."; exit 1 ;;
+        12) echo "No comparison data for commit ${CI_COMMIT_SHA}."
+            echo "Re-run the axe Watcher scan with CI=true, or seed a baseline."
+            # Policy choice: fail. To warn instead, replace with: exit 0
+            exit 1 ;;
+        11) echo "Timed out waiting for the session to finish processing."; exit 1 ;;
+        9)  echo "axe Watcher DevHub returned a server error."; exit 1 ;;
+        2|3) echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags."; exit 1 ;;
+        *)  echo "axe-watcher-results exited with code ${code}."; exit 1 ;;
+      esac
+  artifacts:
+    when: always
+    paths:
+      - result.json

results/shell/gate.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# Provider-agnostic axe Watcher accessibility gate.
+#
+# Retrieves the latest axe Watcher run for a commit and turns the CLI's exit
+# code into a pass/fail decision. Wire this into any CI provider (see the
+# ../github-actions and ../gitlab-ci examples for provider-specific versions).
+#
+# Required environment:
+#   AXE_DEVHUB_API_KEY  axe Developer Hub API key (store as a CI secret).
+#   AXE_PROJECT_ID      Project UUID.
+#   COMMIT_SHA          Commit to gate on. In GitHub Actions on pull_request use
+#                       github.event.pull_request.head.sha (github.sha is the
+#                       merge commit, which Watcher did not scan); GitLab
+#                       $CI_COMMIT_SHA; CircleCI $CIRCLE_SHA1; Jenkins $GIT_COMMIT.
+#
+# This gate resolves the run by COMMIT SHA — the identifier CI hands you for
+# free. `sessions get` also accepts a session UUID; to gate on or fetch a
+# specific session instead, see session-results.sh.
+
+set -euo pipefail
+
+: "${AXE_DEVHUB_API_KEY:?set AXE_DEVHUB_API_KEY (store it as a CI secret)}"
+: "${AXE_PROJECT_ID:?set AXE_PROJECT_ID}"
+: "${COMMIT_SHA:?set COMMIT_SHA to the commit you want to gate on}"
+
+# Capture the exit code rather than letting a non-zero result abort the script,
+# so each meaningful outcome gets a clear message instead of a generic failure.
+set +e
+axe-watcher-results sessions get "$AXE_PROJECT_ID" "$COMMIT_SHA" --format=json >result.json
+code=$?
+set -e
+
+cat result.json
+
+case "$code" in
+  0)
+    echo "Accessibility gate passed."
+    ;;
+  10)
+    echo "Accessibility threshold exceeded — failing the build."
+    exit 1
+    ;;
+  12)
+    echo "No comparison data for commit ${COMMIT_SHA}."
+    echo "Recover by re-running the axe Watcher scan with CI=true, or by seeding a"
+    echo "baseline (a prior scan on this commit, or any earlier commit on the branch)."
+    # Policy choice: fail the build. To warn instead, replace with: exit 0
+    exit 1
+    ;;
+  11)
+    echo "Timed out waiting for the session to finish processing."
+    exit 1
+    ;;
+  9)
+    echo "axe Watcher DevHub returned a server error."
+    exit 1
+    ;;
+  2 | 3)
+    echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags."
+    exit 1
+    ;;
+  *)
+    echo "axe-watcher-results exited with code ${code}."
+    exit 1
+    ;;
+esac

results/shell/session-results.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# Retrieve (and gate on) an axe Watcher run by SESSION ID.
+#
+# `sessions get` accepts a session UUID as well as a commit SHA. Use this when
+# a step already knows the exact session to check — e.g. a Watcher scan step
+# that emits its session ID — rather than resolving one from the commit. The
+# exit-code gating is identical to the commit-SHA path (see gate.sh).
+#
+# Required environment:
+#   AXE_DEVHUB_API_KEY  axe Developer Hub API key (store as a CI secret).
+#   AXE_PROJECT_ID      Project UUID.
+#   SESSION_ID          Session UUID to retrieve.
+
+set -euo pipefail
+
+: "${AXE_DEVHUB_API_KEY:?set AXE_DEVHUB_API_KEY (store it as a CI secret)}"
+: "${AXE_PROJECT_ID:?set AXE_PROJECT_ID}"
+: "${SESSION_ID:?set SESSION_ID to the session UUID to retrieve}"
+
+# The summary report drives the pass/fail gate. Capture the exit code so each
+# outcome gets a clear message instead of a generic failure.
+set +e
+axe-watcher-results sessions get "$AXE_PROJECT_ID" "$SESSION_ID" --format=json >summary.json
+code=$?
+set -e
+
+cat summary.json
+
+case "$code" in
+  0)
+    echo "Accessibility gate passed."
+    ;;
+  10)
+    echo "Accessibility threshold exceeded — failing the build."
+    exit 1
+    ;;
+  12)
+    echo "No comparison data for session ${SESSION_ID}."
+    echo "Recover by re-running the axe Watcher scan with CI=true, or by seeding a baseline."
+    # Policy choice: fail the build. To warn instead, replace with: exit 0
+    exit 1
+    ;;
+  11)
+    echo "Timed out waiting for the session to finish processing."
+    exit 1
+    ;;
+  9)
+    echo "axe Watcher DevHub returned a server error."
+    exit 1
+    ;;
+  2 | 3)
+    echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags."
+    exit 1
+    ;;
+  *)
+    echo "axe-watcher-results exited with code ${code}."
+    exit 1
+    ;;
+esac
+
+# Optional: archive the complete result set. --detail=full streams the raw axe
+# Common Export Format document to stdout and ignores --format. This is a
+# best-effort extra: the gate already passed above, so a transient failure
+# fetching the full document must not fail the build.
+if axe-watcher-results sessions get "$AXE_PROJECT_ID" "$SESSION_ID" --detail=full >results-full.json; then
+  echo "Wrote results-full.json"
+else
+  echo "Warning: could not fetch the full result set; skipping results-full.json." >&2
+fi