diff --git a/results/README.md b/results/README.md new file mode 100644 index 0000000..af2492e --- /dev/null +++ b/results/README.md @@ -0,0 +1,113 @@ +# Gating CI on axe Watcher results + +The framework examples in this repo (see [`js/`](../js) and [`java/`](../java)) +show how to **run** an axe Watcher scan. These examples are one **optional** +way to work with a run's results afterward: pulling them into CI with the +[`axe-watcher-results`](https://github.com/dequelabs/axe-watcher-results) CLI +and turning them into a **pass/fail build gate**, so a build can fail on an +accessibility regression. + +| Provider | Example | +| -------------------------- | -------------------------------------------------------------- | +| GitHub Actions | [`github-actions/a11y-gate.yml`](github-actions/a11y-gate.yml) | +| GitLab CI | [`gitlab-ci/.gitlab-ci.yml`](gitlab-ci/.gitlab-ci.yml) | +| Any (shell) | [`shell/gate.sh`](shell/gate.sh) | +| Any (shell), by session ID | [`shell/session-results.sh`](shell/session-results.sh) | + +## The gating model + +The CLI retrieves the axe Watcher run for a commit and **reflects the outcome +in its exit code** — so a CI step passes or fails based on that code: + +```sh +axe-watcher-results sessions get "$AXE_PROJECT_ID" "$COMMIT_SHA" --format=json +``` + +Each example captures the exit code, prints a clear message per outcome, and +archives the JSON output as a build artifact. + +> The provider examples gate on a **commit SHA** because that is the +> identifier CI already provides (GitHub `github.event.pull_request.head.sha`, +> GitLab `$CI_COMMIT_SHA`). `sessions get` also accepts a **session UUID** — +> see [By session ID](#by-session-id). + +## By session ID + +`sessions get` accepts a **session UUID** as well as a commit SHA. This is +useful when a step already knows the exact session to check — for example a +scan step that emits its session ID — so you gate on that session directly +instead of resolving one from the commit: + +```sh +axe-watcher-results sessions get "$AXE_PROJECT_ID" "$SESSION_ID" --format=json +``` + +The exit-code gating is identical to the commit-SHA path. Two session-only +options: + +- `--detail=summary` (default) — the counts report, rendered per `--format`. +- `--detail=full` — streams the raw axe Common Export Format JSON to stdout + (ignores `--format`), handy to archive the complete result set. + +See [`shell/session-results.sh`](shell/session-results.sh) for a runnable +version. The provider examples above use a commit SHA because CI supplies one +automatically; swap in a session UUID wherever you have one. + +## Exit codes + +| Code | Meaning | How the examples treat it | +| ---- | ----------------------------------- | ------------------------- | +| 0 | Success (under threshold) | Pass the step | +| 2 | Missing required argument / env var | Fail (misconfiguration) | +| 3 | Invalid argument value | Fail (misconfiguration) | +| 9 | Server error from DevHub | Fail, surfaced clearly | +| 10 | Accessibility threshold exceeded | **Fail — the core gate** | +| 11 | Session polling timed out | Fail, surfaced clearly | +| 12 | No comparison data for this commit | Policy choice (see below) | + +**Exit 12 — no comparison data.** The commit has scan results but the server +has nothing to diff them against. The examples **fail by default** and print +how to recover: + +- re-run the axe Watcher scan with `CI=true` (marks the session canonical, so + it self-compares), or +- seed a baseline by scanning this commit again, or any earlier commit on the + same branch. + +Each example shows, in a comment, how to **warn instead of fail** (exit 0) if +your team prefers a softer policy while baselines are being established. + +## Installing the CLI + +Download the released binary for your runner's OS/arch from Agora. Each version +folder holds one binary per platform — +`axe-watcher-results////axe-watcher-results` (`.exe` on +Windows) — for `linux/amd64`, `linux/arm64`, `darwin/arm64`, and +`windows/amd64`, alongside a `SHA256SUMS` manifest to verify the download and +the generated `usage.md`: + +```sh +# TODO: set AGORA_BASE_URL to the published Agora repository base once the +# publish location is finalized (dequelabs/axe-watcher-results#36). +AGORA_BASE_URL="https://agora.dequecloud.com/artifactory/" +CLI_VERSION="1.0.0" # the axe-watcher-results release to install +os="$(uname -s | tr '[:upper:]' '[:lower:]')" # linux or darwin +arch="$(uname -m)" +case "$arch" in x86_64) arch=amd64 ;; aarch64) arch=arm64 ;; esac + +curl -fsSL \ + "$AGORA_BASE_URL/axe-watcher-results/$CLI_VERSION/$os/$arch/axe-watcher-results" \ + -o axe-watcher-results +chmod +x axe-watcher-results +# Then put ./axe-watcher-results on your PATH. The CI examples do this per +# provider — e.g. the GitHub Actions example installs into "$RUNNER_TEMP/bin" +# and appends it to "$GITHUB_PATH". +``` + +## Secrets + +`AXE_DEVHUB_API_KEY` is your axe Developer Hub API key, available from your +axe account settings — always source it from your provider's secret store — +GitHub Actions `secrets`, GitLab **masked** (and protected) CI/CD variables — +never inline it in the pipeline file. `AXE_PROJECT_ID` is the project UUID and +can be a plain variable. diff --git a/results/github-actions/a11y-gate.yml b/results/github-actions/a11y-gate.yml new file mode 100644 index 0000000..f2ae8d5 --- /dev/null +++ b/results/github-actions/a11y-gate.yml @@ -0,0 +1,71 @@ +# axe Watcher accessibility gate for GitHub Actions. +# +# Prerequisites (repository settings): +# - Secret AXE_DEVHUB_API_KEY — your axe Developer Hub API key. +# - Variable AXE_PROJECT_ID — the project UUID. +name: Accessibility Gate +on: + pull_request: + +# The gate needs no repository write access; keep permissions minimal. +permissions: + contents: read + +jobs: + a11y-gate: + name: axe Watcher gate + runs-on: ubuntu-latest + steps: + # Download the released CLI binary for this runner (ubuntu-latest → + # linux/amd64) from Agora. + # TODO: set AGORA_BASE_URL to the published Agora repository base once + # the publish location is finalized (dequelabs/axe-watcher-results#36). + - name: Install axe-watcher-results + env: + AGORA_BASE_URL: https://agora.dequecloud.com/artifactory/ + CLI_VERSION: 1.0.0 # the axe-watcher-results release to install + run: | + mkdir -p "$RUNNER_TEMP/bin" + curl -fsSL \ + "$AGORA_BASE_URL/axe-watcher-results/$CLI_VERSION/linux/amd64/axe-watcher-results" \ + -o "$RUNNER_TEMP/bin/axe-watcher-results" + chmod +x "$RUNNER_TEMP/bin/axe-watcher-results" + echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH" + + - name: Gate on the accessibility threshold + env: + AXE_DEVHUB_API_KEY: ${{ secrets.AXE_DEVHUB_API_KEY }} + AXE_PROJECT_ID: ${{ vars.AXE_PROJECT_ID }} + # On pull_request, github.sha is the merge commit — use the PR head + # SHA, which is the commit axe Watcher actually scanned. + COMMIT_SHA: ${{ github.event.pull_request.head.sha }} + run: | + # Capture the exit code so each outcome gets a clear message. + set +e + axe-watcher-results sessions get "$AXE_PROJECT_ID" "$COMMIT_SHA" --format=json >result.json + code=$? + set -e + + cat result.json + + case "$code" in + 0) echo "Accessibility gate passed." ;; + 10) echo "Accessibility threshold exceeded — failing the build."; exit 1 ;; + 12) echo "No comparison data for commit ${COMMIT_SHA}." + echo "Re-run the axe Watcher scan with CI=true, or seed a baseline." + # Policy choice: fail. To warn instead, replace with: exit 0 + exit 1 ;; + 11) echo "Timed out waiting for the session to finish processing."; exit 1 ;; + 9) echo "axe Watcher DevHub returned a server error."; exit 1 ;; + 2|3) echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags."; exit 1 ;; + *) echo "axe-watcher-results exited with code ${code}."; exit 1 ;; + esac + + # Archive the JSON result for the build logs, even when the gate fails. + - name: Upload result JSON + if: always() + uses: actions/upload-artifact@v4 + with: + name: axe-watcher-result + path: result.json + if-no-files-found: warn diff --git a/results/gitlab-ci/.gitlab-ci.yml b/results/gitlab-ci/.gitlab-ci.yml new file mode 100644 index 0000000..5b0feb3 --- /dev/null +++ b/results/gitlab-ci/.gitlab-ci.yml @@ -0,0 +1,43 @@ +# axe Watcher accessibility gate for GitLab CI. +# +# Prerequisites (Settings > CI/CD > Variables): +# - AXE_DEVHUB_API_KEY — your axe Developer Hub API key. Mark it Masked +# (and Protected). Never inline it here. +# - AXE_PROJECT_ID — the project UUID (a plain variable is fine). +accessibility-gate: + # A small image with curl is all that is needed to fetch the released binary. + image: alpine:3 + # TODO: set AGORA_BASE_URL to the published Agora repository base once the + # publish location is finalized (dequelabs/axe-watcher-results#36). + variables: + AGORA_BASE_URL: https://agora.dequecloud.com/artifactory/ + CLI_VERSION: '1.0.0' # the axe-watcher-results release to install + script: + - apk add --no-cache curl + - curl -fsSL "$AGORA_BASE_URL/axe-watcher-results/$CLI_VERSION/linux/amd64/axe-watcher-results" -o /usr/local/bin/axe-watcher-results + - chmod +x /usr/local/bin/axe-watcher-results + - | + # Capture the exit code so each outcome gets a clear message. + set +e + axe-watcher-results sessions get "$AXE_PROJECT_ID" "$CI_COMMIT_SHA" --format=json >result.json + code=$? + set -e + + cat result.json + + case "$code" in + 0) echo "Accessibility gate passed." ;; + 10) echo "Accessibility threshold exceeded — failing the build."; exit 1 ;; + 12) echo "No comparison data for commit ${CI_COMMIT_SHA}." + echo "Re-run the axe Watcher scan with CI=true, or seed a baseline." + # Policy choice: fail. To warn instead, replace with: exit 0 + exit 1 ;; + 11) echo "Timed out waiting for the session to finish processing."; exit 1 ;; + 9) echo "axe Watcher DevHub returned a server error."; exit 1 ;; + 2|3) echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags."; exit 1 ;; + *) echo "axe-watcher-results exited with code ${code}."; exit 1 ;; + esac + artifacts: + when: always + paths: + - result.json diff --git a/results/shell/gate.sh b/results/shell/gate.sh new file mode 100755 index 0000000..5dc0aaa --- /dev/null +++ b/results/shell/gate.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# Provider-agnostic axe Watcher accessibility gate. +# +# Retrieves the latest axe Watcher run for a commit and turns the CLI's exit +# code into a pass/fail decision. Wire this into any CI provider (see the +# ../github-actions and ../gitlab-ci examples for provider-specific versions). +# +# Required environment: +# AXE_DEVHUB_API_KEY axe Developer Hub API key (store as a CI secret). +# AXE_PROJECT_ID Project UUID. +# COMMIT_SHA Commit to gate on. In GitHub Actions on pull_request use +# github.event.pull_request.head.sha (github.sha is the +# merge commit, which Watcher did not scan); GitLab +# $CI_COMMIT_SHA; CircleCI $CIRCLE_SHA1; Jenkins $GIT_COMMIT. +# +# This gate resolves the run by COMMIT SHA — the identifier CI hands you for +# free. `sessions get` also accepts a session UUID; to gate on or fetch a +# specific session instead, see session-results.sh. + +set -euo pipefail + +: "${AXE_DEVHUB_API_KEY:?set AXE_DEVHUB_API_KEY (store it as a CI secret)}" +: "${AXE_PROJECT_ID:?set AXE_PROJECT_ID}" +: "${COMMIT_SHA:?set COMMIT_SHA to the commit you want to gate on}" + +# Capture the exit code rather than letting a non-zero result abort the script, +# so each meaningful outcome gets a clear message instead of a generic failure. +set +e +axe-watcher-results sessions get "$AXE_PROJECT_ID" "$COMMIT_SHA" --format=json >result.json +code=$? +set -e + +cat result.json + +case "$code" in + 0) + echo "Accessibility gate passed." + ;; + 10) + echo "Accessibility threshold exceeded — failing the build." + exit 1 + ;; + 12) + echo "No comparison data for commit ${COMMIT_SHA}." + echo "Recover by re-running the axe Watcher scan with CI=true, or by seeding a" + echo "baseline (a prior scan on this commit, or any earlier commit on the branch)." + # Policy choice: fail the build. To warn instead, replace with: exit 0 + exit 1 + ;; + 11) + echo "Timed out waiting for the session to finish processing." + exit 1 + ;; + 9) + echo "axe Watcher DevHub returned a server error." + exit 1 + ;; + 2 | 3) + echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags." + exit 1 + ;; + *) + echo "axe-watcher-results exited with code ${code}." + exit 1 + ;; +esac diff --git a/results/shell/session-results.sh b/results/shell/session-results.sh new file mode 100755 index 0000000..83c8636 --- /dev/null +++ b/results/shell/session-results.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Retrieve (and gate on) an axe Watcher run by SESSION ID. +# +# `sessions get` accepts a session UUID as well as a commit SHA. Use this when +# a step already knows the exact session to check — e.g. a Watcher scan step +# that emits its session ID — rather than resolving one from the commit. The +# exit-code gating is identical to the commit-SHA path (see gate.sh). +# +# Required environment: +# AXE_DEVHUB_API_KEY axe Developer Hub API key (store as a CI secret). +# AXE_PROJECT_ID Project UUID. +# SESSION_ID Session UUID to retrieve. + +set -euo pipefail + +: "${AXE_DEVHUB_API_KEY:?set AXE_DEVHUB_API_KEY (store it as a CI secret)}" +: "${AXE_PROJECT_ID:?set AXE_PROJECT_ID}" +: "${SESSION_ID:?set SESSION_ID to the session UUID to retrieve}" + +# The summary report drives the pass/fail gate. Capture the exit code so each +# outcome gets a clear message instead of a generic failure. +set +e +axe-watcher-results sessions get "$AXE_PROJECT_ID" "$SESSION_ID" --format=json >summary.json +code=$? +set -e + +cat summary.json + +case "$code" in + 0) + echo "Accessibility gate passed." + ;; + 10) + echo "Accessibility threshold exceeded — failing the build." + exit 1 + ;; + 12) + echo "No comparison data for session ${SESSION_ID}." + echo "Recover by re-running the axe Watcher scan with CI=true, or by seeding a baseline." + # Policy choice: fail the build. To warn instead, replace with: exit 0 + exit 1 + ;; + 11) + echo "Timed out waiting for the session to finish processing." + exit 1 + ;; + 9) + echo "axe Watcher DevHub returned a server error." + exit 1 + ;; + 2 | 3) + echo "Configuration error — check AXE_DEVHUB_API_KEY, AXE_PROJECT_ID, and flags." + exit 1 + ;; + *) + echo "axe-watcher-results exited with code ${code}." + exit 1 + ;; +esac + +# Optional: archive the complete result set. --detail=full streams the raw axe +# Common Export Format document to stdout and ignores --format. This is a +# best-effort extra: the gate already passed above, so a transient failure +# fetching the full document must not fail the build. +if axe-watcher-results sessions get "$AXE_PROJECT_ID" "$SESSION_ID" --detail=full >results-full.json; then + echo "Wrote results-full.json" +else + echo "Warning: could not fetch the full result set; skipping results-full.json." >&2 +fi