diff --git a/.agents/skills/fix-latest-deps-pr/SKILL.md b/.agents/skills/fix-latest-deps-pr/SKILL.md new file mode 100644 index 00000000000..a57c35aef8b --- /dev/null +++ b/.agents/skills/fix-latest-deps-pr/SKILL.md @@ -0,0 +1,289 @@ +--- +name: fix-latest-deps-pr +description: >- + Triage and unblock the weekly "Update Gradle dependencies" PR when GitLab CI is red from `latestDepTest` breakages. + Use when asked to "fix the latest deps PR", "unblock the gradle dependencies PR", "fix update-gradle-dependencies", or + when given a GitLab pipeline id + PR number. Rolls back the conflicting module lockfiles to make CI green, then + optionally ships per-module real fixes as separate PRs. +user-invocable: true +--- + +# Fix "Update Gradle dependencies" PR + +The weekly GitHub Action `.github/workflows/update-gradle-dependencies.yaml` bumps all latest dependencies and opens up +to two PRs (core + instrumentation). These PRs can go red on GitLab CI because a newly-updated *latest* dependency is +incompatible with current `dd-trace-java` code — the failures surface as `latestDepTest` task failures. + +This skill has two phases: + +1. **Unblock (always):** roll back only the `gradle.lockfile`s of modules whose `latestDep*Test` failed, one commit per + module, then push once. This restores CI so the (still-valuable) lockfile updates for the other modules can merge. +2. **Real fix (opt-in, per module):** actually make the code compatible with the new dependency version, verified + locally, shipped as a separate PR off `master`. + +Only failures of a module's **latest-dep source set** are in scope — its `latestDep*Test` suites *and* their +compile/resolution tasks (e.g. `compileLatestDep*`). Ignore all other red jobs (flaky, infra, unrelated test failures) +— do not touch them. + +--- + +## Prerequisites + +Require `glab`, `gh`, and `jq`. If any is missing or unauthenticated, report exactly what's missing and stop — don't +install it yourself or work around it; print the install command so the user can run it manually. + +- **`glab` (GitLab CLI) — required, authenticated.** `glab auth status` must be green for `gitlab.ddbuild.io`. If it's + missing, tell the user to install it manually (`brew install glab`, or the platform package manager). +- **GitHub CLI (`gh`) — required, authenticated.** Resolves, checks out, and opens PRs. Run `gh auth status` to verify. +- **`jq` — required.** Every GitLab API call is parsed with it; confirm it is on `PATH`. +- **Git remote `origin` with `master` — required.** The rollback baseline and the Phase 3 branch base come from the + latest `origin/master` — always `git fetch origin master` first so they aren't computed against a stale local ref. +- **Module-specific credentials — conditional.** Some optional instrumentations are excluded from the Gradle build + unless a property is set, so their project path is unresolvable — e.g. `akka-http-10.6` needs + `ORG_GRADLE_PROJECT_akkaRepositoryToken`. Skip such a module with a note to the user rather than assuming the mapping + is wrong. + +Every `glab api` call below uses two fixed values: `--hostname gitlab.ddbuild.io` (the remote is GitHub, so `glab` +can't infer the host) and the project path `DataDog%2Fapm-reliability%2Fdd-trace-java` (use exactly this — the shorter +`DataDog/dd-trace-java` only 301-redirects and breaks `glab`). + +--- + +## Phase 0 — Preflight + +1. **Collect inputs.** Ask the user for the **GitLab pipeline id** and the **PR number**. + +2. **Resolve the PR.** + ```bash + gh pr view --json number,headRefName,headRefOid,url,baseRefName,title + ``` + Capture `headRefName` (branch) and `headRefOid` (head SHA). Sanity-check that `baseRefName` is `master` and the + title/branch looks like the weekly dependency-update PR; if not, STOP and tell the user this isn't the PR the skill + handles (the whole skill assumes an `origin/master` baseline and lockfile-only diffs). + +3. **Record the broken head SHA** as session context — call it `ORIG_PR_HEAD`. Substitute the literal 40-char SHA into + every command below rather than a shell variable (each command may run in a separate shell). Phase 3 needs it to + retrieve the *broken* lockfiles after Phase 2 rolls them back. + +4. **Check out the PR branch at its head.** + +- If the worktree is **dirty**, STOP and ask the user to commit/stash/discard — this skill rewrites lockfiles in place + and a checkout could clobber uncommitted work. +- If clean and not already on the PR branch, run `gh pr checkout ` and tell the user you switched them. +- Verify `HEAD`, and if it doesn't match, sync to the PR head (with the user's OK, since `reset --hard` moves the branch + and drops any local commits above it): + ```bash + git rev-parse HEAD # must equal ORIG_PR_HEAD + git fetch origin && git reset --hard # only if HEAD != ORIG_PR_HEAD + ``` + +Do not triage or commit until `HEAD` equals the PR head — otherwise Phase 2 builds commits on the wrong tree. + +5. **Sync the master reference** (rollback + Phase 2/3 base): + ```bash + git fetch origin master + ``` + +--- + +## Phase 1 — Triage failed latestDep modules + +1. **Enumerate the failed jobs** for the given pipeline. First confirm the pipeline is actually the PR's — its SHA must + equal `ORIG_PR_HEAD`, or a wrong/stale id would roll back unrelated lockfiles; STOP if it doesn't match. Then list + the failed jobs (quote the path — zsh globs the `?`; `--paginate` to span all pages): + ```bash + PROJ=DataDog%2Fapm-reliability%2Fdd-trace-java + PIPE= + glab api --hostname gitlab.ddbuild.io "projects/$PROJ/pipelines/$PIPE" | jq -er .sha # must equal ORIG_PR_HEAD + glab api --hostname gitlab.ddbuild.io --paginate \ + "projects/$PROJ/pipelines/$PIPE/jobs?scope=failed&per_page=100" \ + | jq -r '.[] | "\(.id)\t\(.name)\t\(.status)"' + ``` + +2. **Extract failing Gradle tasks** from each failed job's full `/trace` (the `Execution failed for task '…'` line is + mid-log, not in the tail). Fetch into a scratch dir; abort if any trace can't be fetched, or a broken module goes + unidentified: + ```bash + PROJ=DataDog%2Fapm-reliability%2Fdd-trace-java + LOGDIR=$(mktemp -d); trap 'rm -rf "$LOGDIR"' EXIT # don't leave internal CI logs lying around + JOBS=( ...) + for J in "${JOBS[@]}"; do + glab api --hostname gitlab.ddbuild.io "projects/$PROJ/jobs/$J/trace" > "$LOGDIR/$J.log" \ + || { echo "ERROR: trace fetch failed for job $J — abort" >&2; exit 1; } + done + grep -hoE "Execution failed for task '[^']*'" "$LOGDIR"/*.log | sort -u + ``` + +- **Collect only** latest-dep tasks — a break surfaces as either a test-execution failure or a *compile/resolution* + failure of the latest-dep source set (which fails before the test runs). Match any failing task whose name contains + `latestDep` — e.g. `latestDepTest`, `latestDepForkedTest`, `compileLatestDepJava`. **Ignore** everything else + (`:test`, `:forkedTest`, muzzle, infra, …). +- **Record the exact failing task name (s)** per module — don't assume `latestDepTest`; some modules define only + `latestDepForkedTest`, and Phase 3 reuses the real name to verify. +- **Record the JVM** the job ran on (the job name encodes it, e.g. a `j17`/`jdk17` segment). Phase 3 reproduces with + `-PtestJvm=`. If a module failed on more than one JVM, record each — verify them all. +- A single job can contain multiple failing latest-dep tasks — collect them **all**, across all jobs, deduped by module. + +3. **Map each Gradle task path → lockfile.** Strip the trailing `:`, convert the remaining `:` to `/`, and + append `/gradle.lockfile` — e.g. `::latestDepTest` → `/gradle.lockfile`. + Verify the derived path exists; if it doesn't (rare Gradle-path/dir mismatch), resolve it with + `./gradlew -q ::properties | grep '^projectDir:'` rather than guessing by leaf name. + Then confirm the PR actually changed this lockfile; if it didn't, the failure is flaky/unrelated (rolling back would + be a no-op) — exclude the module and flag it to the user: + ```bash + BASE=$(git merge-base HEAD origin/master) # HEAD == ORIG_PR_HEAD (verified in Phase 0) + git diff --quiet "$BASE" HEAD -- /gradle.lockfile \ + && echo "UNCHANGED — exclude (flaky/unrelated)" || echo "changed — in scope" + ``` + +4. **Report the triage** to the user before making any change: each failed latestDep module with its Gradle path, + lockfile, failing task (s), and JVM (s). + +--- + +## Phase 2 — Unblock (rollback lockfiles) + +For **each** failed module, create **one commit** rolling its lockfile back to the pre-update state. Do **not** push +between commits. + +**Approval model:** the rollback commits are local and fully reversible, so create them without prompting. The one +action that needs the user's go-ahead is the **push** (step 4) — where changes leave the machine and re-trigger CI. + +1. Roll the module's lockfile back to the pre-update baseline. Run as **one block** — `BASE` must live in the same + shell: + ```bash + BASE=$(git merge-base HEAD origin/master) + M=/gradle.lockfile + if git cat-file -e "$BASE:$M" 2>/dev/null; then + git checkout "$BASE" -- "$M" # existed at BASE (common case — an update) → restore & stage + else + git rm "$M" # created by the update → rolling back means deleting it + fi + ``` + +2. Commit that single module's already-staged change (restore or deletion — no `git add` needed): + ```bash + git commit -m "temporary fix: rolled back conflicting latest dependencies for module: to unblock PR merging" + ``` + +3. Repeat for every failed module — one commit each. + +4. **Verify, then push once** (after all commits exist). One diff per module confirms the lockfile now matches the + baseline — this covers both cases (a restored file has no diff vs `BASE`; a correctly-deleted new file is absent in + both `BASE` and `HEAD`, so also no diff): + ```bash + BASE=$(git merge-base HEAD origin/master) + git diff --quiet "$BASE" HEAD -- /gradle.lockfile \ + && echo "matches baseline" || echo "DIFF — investigate before pushing" + ``` + Fix anything flagged. Then, with the user's go-ahead, push **explicitly** to the PR branch (`headRefName` from Phase + 0) so a missing/incorrect upstream can't send commits elsewhere: + ```bash + git push origin HEAD: + ``` + +One push (not per commit) triggers a single CI run. Report the pushed commits and remind the user CI will re-run. + +--- + +## Phase 3 — Real fix (opt-in, per module) + +After unblocking, **ask** the user whether to create separate real-fix PRs (one per failed module). If no, stop and hand +them the module list. If yes, work modules **one at a time** — fully finish and verify one before starting the next. + +For each module: + +1. **Fresh branch off master.** Name it from the **sanitized fully-qualified Gradle path** (strip the leading `:`, + replace every `:` with `-`), *not* the leaf — two modules can share a leaf (e.g. `grpc-1.5`) and collide: + ```bash + git fetch origin master + git checkout -b fix/latest-dep- origin/master + ``` + +2. **Reproduce the failure** by putting the module's lockfile into the *broken* PR-head state. Substitute the literal + `ORIG_PR_HEAD` SHA. The update regenerates every lockfile and can *delete* one, so mirror whichever state it has: + ```bash + if git cat-file -e ":/gradle.lockfile" 2>/dev/null; then + git checkout -- /gradle.lockfile # present at PR head → restore + else + git rm /gradle.lockfile # absent at PR head → reproduce the deletion + fi + ``` + +3. **Research the breaking change:** + +- Full CI logs via the `/trace` fetch in Phase 1 step 2; download `reports.tar` for thread/heap dumps if it's a + hang/crash. +- Read the failing module's source and tests. +- Diff the conflicting dependency old vs new: release notes, changelog, tags/diffs, decompile the new jar if needed. Use + web-fetch / web-search for release notes and upstream docs. + +4. **Implement the fix** so the module is compatible with the new version. Usually production/test source, but often + also module-local `build.gradle` changes (constraints, forced versions, source-set config) or test + fixtures/resources — keep those too. If the fix genuinely requires changes **outside** the failing module (a shared + helper, a related version module), that's allowed with clear justification, but you must extend verification (step 5) + to every module you touched. Match surrounding code style; follow repo test conventions. + +5. **Verify locally — the latest-dep suite must go green WITHOUT breaking the base test set** (a separate source set). + Discover what the module defines (don't assume `latestDepTest`): + ```bash + ./gradlew ::tasks --group=verification + ``` + Then, on the JVM (s) you recorded in triage (run each one that failed), require all to pass: + ```bash + ./gradlew :: -PtestJvm= + ./gradlew ::test -PtestJvm= + ``` + Also run any other verification task the module defines, and the `test` (and latest-dep) tasks of any **other** + module you changed. Skipping `-PtestJvm` runs your default JVM and can falsely pass a JVM-specific break. Do not + proceed while anything is red. + +6. **Only when everything is green**, commit, push the branch, and open a **draft** PR off `master`. + + **Write a real, filled-in PR description — never a stub.** Base it on `.github/pull_request_template.md` (read that + file, fill each section from what you found and did this run): **What Does This Do** (the change + dependency version + it targets), **Motivation** (which `latestDep` upgrade broke and the API/behavior change that caused it), + **Additional Notes** (tests run, links to release notes/upstream diff; omit if empty), and the **Contributor + Checklist**. + + **Stage only the intended files** — never `git add -A`. The lockfile's step-2 state (restore *or* deletion) is + already staged, so stage just the files you changed here and re-add the lockfile **only if it still exists** + (re-adding a deleted path errors). **Confirm with the user before committing** — show the staged diff and the PR + title/body, and get an explicit go-ahead (this pushes a branch and opens an external PR). Then: + ```bash + git add + [ -e /gradle.lockfile ] && git add -- /gradle.lockfile # deletion already staged from step 2 + git status && git diff --cached # review exactly what will be committed — show this to the user + # → get the user's go-ahead here, then: + git commit -m " version>" + git push -u origin fix/latest-dep- + gh pr create --draft --base master \ + --title "" \ + --label "tag: ai generated" \ + --label "tag: dependencies" \ + --label "" \ + --label "" \ + --body "$(cat <<'EOF' + + EOF + )" + ``` + Per repo PR conventions: draft first; always include `tag: ai generated`, at least one `comp:`/`inst:` label, and one + `type:` label. + +7. Return to a clean state for the next module: + ```bash + git checkout + ``` + +When done, summarize: unblock commits pushed, and for each module either the fix PR URL or that it was skipped. + +--- + +## Guardrails + +- **Scope:** only modules whose latest-dep source set failed. Phase 2 changes **only** those modules' `gradle.lockfile`s + — never source code, never another module. Phase 3 may extend beyond the failed module only with clear justification, + and verification must then cover every module you touched. +- **Approvals:** get the user's go-ahead before the Phase 2 push and before committing/opening any Phase 3 PR. +- Keep Gradle runs sequential. diff --git a/.claude/skills/fix-latest-deps-pr b/.claude/skills/fix-latest-deps-pr new file mode 120000 index 00000000000..798ea54f823 --- /dev/null +++ b/.claude/skills/fix-latest-deps-pr @@ -0,0 +1 @@ +../../.agents/skills/fix-latest-deps-pr \ No newline at end of file