-
Notifications
You must be signed in to change notification settings - Fork 2
ci(eval): add functional tests for triage agent #31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ralphbean
wants to merge
1
commit into
main
Choose a base branch
from
functional-tests-triage
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| name: Check E2E Authorization | ||
| description: >- | ||
| Authorize PR-triggered e2e runs for trusted authors or a fresh ok-to-test label. | ||
| Removes stale ok-to-test labels and posts a sticky PR comment when unauthorized. | ||
|
|
||
| inputs: | ||
| pr_number: | ||
| description: Pull request number to authorize | ||
| required: true | ||
| repository: | ||
| description: Repository in owner/name form | ||
| required: true | ||
| pr_updated_at: | ||
| description: github.event.pull_request.updated_at (server-side push time) | ||
| required: false | ||
| default: "" | ||
| event_action: | ||
| description: github.event.action | ||
| required: false | ||
| default: "" | ||
| pr_author_association: | ||
| description: github.event.pull_request.author_association (frozen event payload) | ||
| required: false | ||
| default: "" | ||
| pr_author_login: | ||
| description: github.event.pull_request.user.login (for collaborator permission API fallback) | ||
| required: false | ||
| default: "" | ||
|
|
||
| outputs: | ||
| authorized: | ||
| description: Whether e2e tests may run for this PR | ||
| value: ${{ steps.check.outputs.authorized }} | ||
| reason: | ||
| description: Authorization outcome reason code | ||
| value: ${{ steps.check.outputs.reason }} | ||
| label_removed: | ||
| description: Whether a stale ok-to-test label was removed | ||
| value: ${{ steps.check.outputs.label_removed }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Check authorization | ||
| id: check | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| PR_NUMBER: ${{ inputs.pr_number }} | ||
| REPOSITORY: ${{ inputs.repository }} | ||
| PR_UPDATED_AT: ${{ inputs.pr_updated_at }} | ||
| EVENT_ACTION: ${{ inputs.event_action }} | ||
| PR_AUTHOR_ASSOCIATION: ${{ inputs.pr_author_association }} | ||
| PR_AUTHOR_LOGIN: ${{ inputs.pr_author_login }} | ||
| run: bash "${GITHUB_WORKSPACE}/.github/scripts/check-e2e-authorization.sh" "${PR_NUMBER}" "${REPOSITORY}" | ||
|
|
||
| - name: Post gate comment | ||
| if: steps.check.outputs.authorized != 'true' | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| REPOSITORY: ${{ inputs.repository }} | ||
| PR_NUMBER: ${{ inputs.pr_number }} | ||
| REASON: ${{ steps.check.outputs.reason }} | ||
| LABEL_REMOVED: ${{ steps.check.outputs.label_removed }} | ||
| BOT_LOGIN: github-actions[bot] | ||
| run: | | ||
| set -euo pipefail | ||
|
|
||
| marker='<!-- e2e-gate -->' | ||
| owner="${REPOSITORY%%/*}" | ||
| repo="${REPOSITORY#*/}" | ||
|
|
||
| body="${marker} | ||
| ## Functional tests did not run | ||
|
|
||
| " | ||
|
|
||
| case "${REASON}" in | ||
| stale_ok_to_test) | ||
| body+="The \`ok-to-test\` label was removed because new commits landed after it was applied. A maintainer must re-apply \`ok-to-test\` after reviewing the latest changes. | ||
| " | ||
| ;; | ||
| error) | ||
| body+="The authorization check failed (GitHub API error). Re-run the workflow when the API is available; if this persists, contact a maintainer. | ||
| " | ||
| ;; | ||
| *) | ||
| body+="Functional tests run automatically for org/repo **members and collaborators** on pull requests. | ||
|
|
||
| For other contributors, a maintainer must add the \`ok-to-test\` label **after** the latest push. | ||
| " | ||
| ;; | ||
| esac | ||
|
|
||
| if [[ "${LABEL_REMOVED}" == "true" ]]; then | ||
| body+=" | ||
| > **Note:** \`ok-to-test\` was cleared due to new commits. | ||
| " | ||
| fi | ||
|
|
||
| existing_id="$(gh api "repos/${owner}/${repo}/issues/${PR_NUMBER}/comments" --paginate \ | ||
| | jq -r --arg login "${BOT_LOGIN}" --arg marker "${marker}" \ | ||
| '.[] | select(.user.login == $login and (.body | contains($marker))) | .id' \ | ||
| | head -n1 || true)" | ||
|
|
||
| if [[ -n "${existing_id}" ]]; then | ||
| jq -n --arg body "${body}" '{body: $body}' | \ | ||
| gh api -X PATCH "repos/${owner}/${repo}/issues/comments/${existing_id}" --input - >/dev/null | ||
| else | ||
| jq -n --arg body "${body}" '{body: $body}' | \ | ||
| gh api -X POST "repos/${owner}/${repo}/issues/${PR_NUMBER}/comments" --input - >/dev/null | ||
| fi | ||
|
|
||
| - name: Clear gate comment | ||
| if: steps.check.outputs.authorized == 'true' | ||
| shell: bash | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| REPOSITORY: ${{ inputs.repository }} | ||
| PR_NUMBER: ${{ inputs.pr_number }} | ||
| BOT_LOGIN: github-actions[bot] | ||
| run: | | ||
| set -euo pipefail | ||
|
|
||
| marker='<!-- e2e-gate -->' | ||
| owner="${REPOSITORY%%/*}" | ||
| repo="${REPOSITORY#*/}" | ||
|
|
||
| existing_id="$(gh api "repos/${owner}/${repo}/issues/${PR_NUMBER}/comments" --paginate \ | ||
| | jq -r --arg login "${BOT_LOGIN}" --arg marker "${marker}" \ | ||
| '.[] | select(.user.login == $login and (.body | contains($marker))) | .id' \ | ||
| | head -n1 || true)" | ||
|
|
||
| if [[ -n "${existing_id}" ]]; then | ||
| body="${marker} | ||
| ## Functional tests are running | ||
|
|
||
| Authorization passed for this commit. See the **Functional Tests** workflow for results." | ||
| jq -n --arg body "${body}" '{body: $body}' | \ | ||
| gh api -X PATCH "repos/${owner}/${repo}/issues/comments/${existing_id}" --input - >/dev/null | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| #!/usr/bin/env bash | ||
| # check-e2e-authorization.sh — Decide whether a PR may run e2e tests in CI. | ||
| # | ||
| # Authorized when the PR author is OWNER/MEMBER/COLLABORATOR, when the author | ||
| # is a trusted bot, when the collaborator permission API confirms write+ | ||
| # access, or when a fresh ok-to-test label was applied after the latest push. | ||
| # | ||
| # Usage: check-e2e-authorization.sh PR_NUMBER OWNER/REPO | ||
| # | ||
| # Environment (optional, from workflow): | ||
| # PR_AUTHOR_ASSOCIATION — github.event.pull_request.author_association | ||
| # PR_AUTHOR_LOGIN — github.event.pull_request.user.login | ||
| # PR_UPDATED_AT — github.event.pull_request.updated_at | ||
| # EVENT_ACTION — github.event.action | ||
| # | ||
| # Writes authorized, reason, and label_removed to GITHUB_OUTPUT when set. | ||
| # Exits 0 always; callers inspect outputs. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| PR_NUMBER="${1:?PR number required}" | ||
| REPOSITORY="${2:?repository (owner/repo) required}" | ||
|
|
||
| TRUSTED_ASSOCIATIONS="OWNER MEMBER COLLABORATOR" | ||
| OK_TO_TEST_LABEL="ok-to-test" | ||
|
|
||
| write_error_output() { | ||
| echo "check-e2e-authorization: API or script error (see log above)" >&2 | ||
| if [[ -n "${GITHUB_OUTPUT:-}" ]]; then | ||
| { | ||
| echo "authorized=false" | ||
| echo "reason=error" | ||
| echo "label_removed=false" | ||
| } >>"${GITHUB_OUTPUT}" | ||
| fi | ||
| printf 'authorized=false reason=error label_removed=false\n' | ||
| } | ||
|
|
||
| trap 'write_error_output; exit 0' ERR | ||
|
|
||
| is_trusted_author() { | ||
| local assoc="$1" | ||
| case " ${TRUSTED_ASSOCIATIONS} " in | ||
| *" ${assoc} "*) return 0 ;; | ||
| *) return 1 ;; | ||
| esac | ||
| } | ||
|
|
||
| # Fallback: check actor has write+ permission via the collaborator permission | ||
| # API, which correctly resolves org membership regardless of visibility. | ||
| has_write_permission() { | ||
| local username="${1:-}" | ||
| if [[ -z "${username}" ]]; then | ||
| return 1 | ||
| fi | ||
| local perm_json role | ||
| perm_json=$(gh api "repos/${REPOSITORY}/collaborators/${username}/permission" 2>/dev/null) || return 1 | ||
| role=$(jq -r '.role_name' <<<"${perm_json}") || return 1 | ||
| case "${role}" in | ||
| admin|maintain|write) return 0 ;; | ||
| *) return 1 ;; | ||
| esac | ||
| } | ||
|
|
||
| label_removed=false | ||
| authorized=false | ||
| reason="unauthorized" | ||
|
|
||
| # Try the frozen workflow event payload first (fast path). If it reports an | ||
| # untrusted association, has_write_permission falls back to the collaborator | ||
| # permission API which resolves correctly regardless of membership visibility. | ||
| if [[ -n "${PR_AUTHOR_ASSOCIATION:-}" ]]; then | ||
| author_association="${PR_AUTHOR_ASSOCIATION}" | ||
| else | ||
| pr_json="$(gh api "repos/${REPOSITORY}/pulls/${PR_NUMBER}")" | ||
| author_association="$(jq -r '.author_association' <<<"${pr_json}")" | ||
| fi | ||
|
|
||
| if is_trusted_author "${author_association}"; then | ||
| authorized=true | ||
| reason="trusted_author" | ||
| elif has_write_permission "${PR_AUTHOR_LOGIN:-}" 2>/dev/null; then | ||
| authorized=true | ||
| reason="trusted_author" | ||
| else | ||
| pr_json="${pr_json:-$(gh api "repos/${REPOSITORY}/pulls/${PR_NUMBER}")}" | ||
| has_ok_label="$(jq -r --arg label "${OK_TO_TEST_LABEL}" '[.labels[].name] | index($label) != null' <<<"${pr_json}")" | ||
|
|
||
| if [[ "${has_ok_label}" == "true" && "${EVENT_ACTION:-}" == "labeled" ]]; then | ||
| authorized=true | ||
| reason="ok_to_test" | ||
| elif [[ "${has_ok_label}" == "true" ]]; then | ||
| events_json="$(gh api "repos/${REPOSITORY}/issues/${PR_NUMBER}/events" --paginate | jq -s 'add // []')" | ||
| ok_to_test_at="$(jq -r --arg label "${OK_TO_TEST_LABEL}" ' | ||
| [.[] | select(.event == "labeled" and (.label.name // "") == $label) | .created_at] | max // empty | ||
| ' <<<"${events_json}")" | ||
|
|
||
| last_push_at="${PR_UPDATED_AT:-}" | ||
| if [[ -z "${last_push_at}" ]]; then | ||
| last_push_at="$(jq -r '.updated_at // empty' <<<"${pr_json}")" | ||
| fi | ||
|
|
||
| if [[ -n "${ok_to_test_at}" && -n "${last_push_at}" && "${ok_to_test_at}" > "${last_push_at}" ]]; then | ||
| authorized=true | ||
| reason="ok_to_test" | ||
| else | ||
| if [[ "${CHECK_E2E_AUTH_DRY_RUN:-}" != "true" ]]; then | ||
| gh api -X DELETE "repos/${REPOSITORY}/issues/${PR_NUMBER}/labels/${OK_TO_TEST_LABEL}" >/dev/null | ||
| fi | ||
| label_removed=true | ||
| reason="stale_ok_to_test" | ||
| fi | ||
| fi | ||
| fi | ||
|
|
||
| trap - ERR | ||
|
|
||
| if [[ -n "${GITHUB_OUTPUT:-}" ]]; then | ||
| { | ||
| echo "authorized=${authorized}" | ||
| echo "reason=${reason}" | ||
| echo "label_removed=${label_removed}" | ||
| } >>"${GITHUB_OUTPUT}" | ||
| fi | ||
|
|
||
| printf 'authorized=%s reason=%s label_removed=%s\n' "${authorized}" "${reason}" "${label_removed}" | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[low] edge-case
The fallback timestamp comparison uses updated_at (which advances on any PR activity) as a proxy for last push time. In the normal workflow path, PR_UPDATED_AT is always passed from the event payload, so this only affects unexpected invocations.